* This article is based on an actual analysis project; however, in light of the confidentiality agreement (NDA) with the client, while maintaining the framework of the medical and healthcare field, specific details such as disease names and variables have been substantially altered from the actual case. We ask for your understanding in advance.

This case compared and verified, in critically ill sepsis patients admitted to the intensive care unit (ICU), the predictive accuracy of multiple novel blood and urinary biochemical indicators (biomarkers) and a machine-learning-based “integrated prediction score” calculated from the patients’ basic information, for predicting the “early onset of acute kidney injury (AKI).”

In the clinical setting, when there are multiple test indicators for predicting the onset of a disease, it is insufficient merely to evaluate individually whether each indicator “is useful for prediction”; it is necessary to rigorously compare whether “the new indicator is statistically significantly superior to the existing one.” However, when comparing multiple indicators measured simultaneously from the same patient population, a strong correlation arises among the data (a paired data structure), so using the wrong statistical method carries the risk of misjudging the superiority of the tests. Furthermore, repeating round-robin comparisons among multiple indicators markedly increases the probability of a significant difference appearing by chance (a Type I error).

Through this analysis, I aimed to apply an advanced statistical test that can correctly handle the paired data structure obtained from the same patient, and further to control the increase in error accompanying multiple tests using the concept of the false discovery rate (FDR). By combining the precise comparison of the area under the ROC curve (AUC) with multiple-comparison correction that prevents false positives while maintaining power, Dr.DataScience scientifically identified the optimal predictive indicator that can be truly relied upon in the clinical setting, making a substantial contribution to establishing an early-treatment-intervention protocol for critically ill patients.

Background and objective

In this case, what was required was to determine the optimal indicator for accurately predicting, at an extremely early stage before onset, “acute kidney injury (AKI)”—a complication with high lethality that develops following septic shock and the like. At the target medical institution, in addition to the prediction indicator currently in mainstream use, there were a total of five evaluation axes: three trace biochemical indicators newly under consideration for introduction, and a “novel integrated prediction score” automatically calculated from patients’ electronic medical-record information.

The greatest clinical challenge was that, because introducing a new test method or prediction score requires corresponding cost and effort, one had to prove as an objective figure “whether the new indicator truly has discriminative ability surpassing the existing indicators.” Merely lining up and looking at the sensitivity and specificity of each indicator cannot determine whether the difference is statistically meaningful or merely chance variability in sampling.

The main objective of the analysis was to calculate, for all five prediction indicators, the area under the ROC curve (AUC)—an index of the ability to discriminate the presence or absence of AKI onset—and to clarify whether a statistically significant difference exists among the AUCs. Another goal was to perform appropriate multiple-comparison correction based on two different premises (case settings)—”determining superiority among the biochemical indicators,” which is the focus of the study, and “determining superiority among all indicators including the integrated prediction score”—and to derive conclusions based on rigorous evaluation criteria.

Data and variables

This analysis used prospectively collected clinical observational data of several hundred critically ill patients at a particular advanced acute-care medical institution. For all patients, the test values at admission were comprehensively obtained. The main variables analyzed were as follows.

1. • Response variable (outcome): the presence or absence of acute kidney injury (AKI) onset within 48 hours after ICU admission (a binary variable: onset = 1, no onset = 0).
   • Prediction variables (continuous variables): the following five indicators were evaluated as independent predictors.
     • Biochemical indicator A (an existing standard blood test value)
     • Biochemical indicator B (a novel urinary trace-protein indicator)
     • Biochemical indicator C (a specific indicator reflecting early renal tubular injury)
     • Biochemical indicator D (an oxidative-stress marker)
     • Novel integrated prediction score (a continuous score value calculated in advance by combining multiple elements such as age, blood pressure, and heart rate)

Analytical methods

In this case, to compare the discriminative ability of multiple prediction variables obtained from the same patient and to avoid the statistical fallacies accompanying multiple tests, I selected and applied the following extremely rigorous statistical methods.

1. Quantification and statistical comparison of predictive ability
   • Adopted methods: ROC analysis and the DeLong test (two-sided)
     To quantify the predictive (discriminative) ability of each indicator, I created ROC curves plotting the sensitivity and false-positive rate at all thresholds, and estimated their area under the curve (AUC) and 95% confidence intervals. I then adopted the “DeLong test” to test the differences among the calculated AUCs. The indicator groups compared this time were all measured simultaneously from “the same patient population,” so correlations inevitably exist among the indicators. Because the DeLong test can accurately evaluate the differences of such “paired (correlated) ROC curves” with a nonparametric approach that does not assume a normal distribution, I judged it the most suitable method in this analysis.
   • Methods not adopted: the Hanley–McNeil test and the Z-test for two independent groups
     The Hanley–McNeil test is a parametric method requiring the assumption of a normal distribution, and its estimates become unstable for data showing extremely skewed distributions like biochemical indicators, so I refrained from adopting it. Also, a test such as the ordinary Z-test, which stands on the premise that “the two ROC curves are independent and obtained from completely different patient populations,” has the fatal flaw of completely ignoring the within-patient correlation structure, overestimating the standard error, and consequently overlooking significant differences (markedly reducing power), so I clearly excluded it.
2. Controlling the Type I error accompanying multiple comparisons
   • Adopted method: control of the false discovery rate (FDR) by the Benjamini–Hochberg method (BH method)
     In this study, I needed to create pairs in ascending order of AUC and repeat multiple tests (indicator A vs. B, B vs. C, etc.). Performing multiple tests increases “the probability of erroneously judging a significant difference by chance when in truth there is none (a Type I error).” To address this, I adopted the Benjamini–Hochberg method (BH method). The BH method, rather than rejecting all results, can keep the proportion of falsely declared significant results (FDR: False Discovery Rate) below a certain level (0.05 or 0.10) while maintaining high “power” to find truly significant differences. I judged it the correction method with the most reasonable balance for the clinical objective of exploring useful novel indicators.
   • Method not adopted: not performing the Bonferroni method
     The most classical correction method, the Bonferroni method, is a very conservative (overly strict) method that multiplies the p-value of every test by the number of tests. Using this method, for real clinical data with even somewhat limited sample sizes, the risk is extremely high of making an excessive judgment of “no significant difference for everything” (an increase in Type II error) even when a clinically meaningful difference truly exists; I therefore judged it inappropriate for a comparison with exploratory elements like this case. And of course, deriving conclusions from raw p-values alone without performing any multiple-test correction is an act of completely abandoning statistical validity, so I strictly avoided it.

Overview of the main results and clinical considerations

In the analysis of “Case 1,” which compared only the biochemical indicators, I evaluated by the DeLong test whether there was a difference between indicator A, which had the lowest predictive ability (AUC = 0.315), and indicator D, which showed moderate predictive ability (AUC = 0.887); the raw p-value was p = 0.016, and a significant difference was found. Furthermore, when I applied multiple-comparison correction by the BH method to this result, the calculated q-value was q = 0.047, which fell below the predetermined false-discovery-rate threshold (FDR = 0.05), so the robust conclusion was drawn—even after correction—that “indicator D has statistically significantly superior predictive ability to indicator A.”

On the other hand, in the DeLong tests among the top three indicators—indicator D (AUC = 0.887), indicator C (AUC = 0.940), and indicator B (AUC = 0.946)—the raw p-values all exceeded 0.05 (for example, p = 0.057 for indicator D vs. indicator C), and naturally no significant difference was found after BH correction (q > 0.05).

Furthermore, in the analysis of “Case 2,” which added the novel integrated prediction score (AUC = 0.149), the criterion for multiple-comparison correction became stricter due to the increased total number of tests. As a result, in the comparison of indicator A and indicator D, although the raw p-value was p = 0.016, the q-value after BH correction was q = 0.062. This presented a result on a delicate borderline left to clinical judgment: under the extremely strict criterion of FDR = 0.05, “it cannot be definitively asserted that there is a significant difference (reserved),” whereas if the FDR = 0.10 criterion permissible for exploratory indicators is adopted, it is judged “significant.”

From these multifaceted analysis results, an extremely clear overall clinical conclusion was obtained: “Indicators B, C, and D have clearly superior predictive ability compared with indicator A and the novel integrated prediction score, but among these top three indicators there is no significant superiority or inferiority that can be statistically proven.” This enabled the medical institution to decide on a cost-effective introduction plan: it need not introduce all three expensive top indicators, but may select just one that is easiest to operate from the standpoint of ease of measurement and cost.

Dr.DataScience’s contribution

In this case, Dr.DataScience completely plugged the statistical pitfalls lurking in a simple comparison of indicators and created unshakable objective evidence to support an important decision in the medical setting.

1. Selecting the optimal comparison method matched to the data structure
   • Accurately discerning the structural characteristic that “this is paired data obtained from the same patient,” I excluded careless independent-group tests and parametric methods and applied the DeLong test. This succeeded in comparing and measuring the true differences in each indicator’s discriminative ability extremely accurately, neither over- nor under-estimating them.
2. Providing advanced correction techniques that resolve the dilemma of multiple comparisons
   • I appropriately controlled the serious problem of “increased Type I error (spurious significant differences)” that always arises when comparing multiple indicators, using the Benjamini–Hochberg method. By avoiding overly strict methods like the Bonferroni method, I derived scientifically valid conclusions without overlooking true clinical value.
3. Supporting flexible decision-making according to the interpretation criterion (FDR level)
   • Rather than discarding the corrected results into a binary “significant/not significant” dichotomy, I created and presented a detailed evaluation table from the perspectives of both an FDR threshold of 0.05 and 0.10. This provided the client with powerful material for making the most appropriate clinical judgment according to the research objective—”whether to prioritize the rigor of the study or the exploratory value of a new indicator.”