* This article is based on an actual analysis project; however, in light of the confidentiality agreement (NDA) with the client, while maintaining the framework of the medical and healthcare field, specific details such as disease names and variables have been substantially altered from the actual case. We ask for your understanding in advance.

This case verified, using advanced machine-learning algorithms, how accurately the over-time change patterns of subjective symptom evaluations that patients input daily themselves (patient-reported outcomes: PRO), in a remote-monitoring system for chronic heart-failure patients, can predict the future “risk of emergency hospitalization due to acute exacerbation.”

In clinical research, the greatest obstacle when predicting serious adverse health events such as “severe deterioration,” “death,” and “emergency hospitalization” is the “extreme imbalance of the data.” In everyday monitoring data, cases that actually lead to emergency hospitalization account for only a few percent of the whole. Applying general statistical analysis or machine learning as-is to such skewed data produces a fatal flaw: the model achieves a high surface-level accuracy rate merely by predicting that “everyone is safe (will not be hospitalized),” while overlooking the dangerous signs that truly must be found.

Through this analysis, I aimed to combine cutting-edge sampling techniques for resolving extreme data imbalance with an algorithm that automatically extracts the truly important signs from a vast number of variables, and to build a high-accuracy predictive model that incorporates the complex interactions among variables. By placing at its core the random-forest method, which integrates many decision trees rather than a single decision tree, and further performing meticulous tuning through Bayesian optimization, Dr.DataScience provided solid objective evidence toward establishing an “early-warning system” that the clinical setting can truly rely on.

Background and objective

In this case, using the daily questionnaire data of the app (shortness of breath, fatigue, lower-limb edema, quality of sleep, etc.) transmitted daily from chronic heart-failure patients recuperating at home, what was required was to quantitatively identify “patients with what kind of symptom trajectory are prone to causing emergency hospitalization in the near future.”

Conventional alert criteria in the clinical setting tended to depend on a simple single-time-point score evaluation, such as “issue a warning if today’s shortness-of-breath score exceeds a threshold.” However, actual acute exacerbations do not occur suddenly; powerful precursors lurk within dynamic change patterns, such as “fatigue has gradually worsened over the past three days” and “not a transient shortness of breath, but a poor condition continuing for several days.” Because the provided data were daily repeated-record data, I needed to create as many as several hundred patterns of new features (evaluation variables) indicating “the degree of change over the most recent days” and “the continuity of the condition” relative to the time of last confirmation, and to identify the true risk factors among them.

The main objective of the analysis was to build a machine-learning model that can capture, with high accuracy, the extremely small number of “emergency-hospitalization events (positive cases)” among the follow-up data of several thousand patients, and to clarify which pattern of symptom trajectory (e.g., three consecutive days of worsening fatigue) is the most powerful predictor. Also, because in the medical setting too many “false alarms (false positives)” increase the burden on healthcare workers and lead to alerts being ignored, an important challenge was to discern the optimal balance between “recall,” which does not overlook patients in a dangerous state, and “precision,” which indicates the accuracy of the warnings.

Data and variables

This analysis used an integrated database of daily remote-monitoring data and electronic medical records collected over a long period from several thousand patients belonging to a particular wide-area medical network. To perform the analysis, I logically converted all the daily string and graded-rating data into numbers, and extracted and shaped them into a complete dataset with no missing values. The main variables analyzed were as follows.

1. • Response variable (prediction target): the presence or absence of “emergency hospitalization due to acute exacerbation of heart failure” within a certain period from the most recent report (hospitalization = true, no hospitalization = false). * The proportion of hospitalization was about 0.6% of the whole—an extremely imbalanced state.
   • Basic attribute variables (confounders): age, sex, cardiac-function classification (NYHA functional classification), presence of comorbidities, recent blood-test results (such as BNP level), class of prescribed drug, etc.
   • Dynamic features (explanatory variables): a group of dynamic variables newly created from the daily questionnaire scores.
     ・”Amount of change from X days ago” (e.g., if the shortness-of-breath score was 2 three days ago and is 4 today, that change pattern)
     ・”Continuous trajectory” (e.g., a state in which the edema score has continued to worsen for three consecutive days)

Analytical methods

In this case, to prevent the model’s predictive dysfunction caused by the extreme skew of the response variable and to make high-accuracy predictions from combinations of several hundred complex variables, I selected and applied the following advanced machine-learning methods and statistical approaches.

1. Correction of the extreme data imbalance
   • Adopted method: hybrid sampling combining SMOTE-NC and ENN
     To solve the problem of too few hospitalized patients (positive cases), I first adopted “SMOTE-NC (Synthetic Minority Over-sampling Technique for Nominal and Continuous variables).” Rather than simply copying and increasing the small amount of data, this is a technique that learns the data structure of actual hospitalized patients (a distribution based on k-nearest neighbors) and artificially generates logically plausible “virtual hospitalized-patient data” to amplify the minority group. Furthermore, to the majority group of non-hospitalized patients, I applied a method called “ENN (Edited Nearest Neighbours)” to selectively delete “confusing majority data near the boundary” that closely resemble the hospitalized-patient data and confuse the classification algorithm. This built a high-quality training dataset in which the characteristics of each class are clearly separated.
   • Methods not adopted: simple random over-sampling or under-sampling
     The method of simply copying and increasing the minority data strongly causes “overfitting,” in which the model excessively memorizes only particular patient data, so I refrained from adopting it. Also, the method of randomly cutting the majority data to match the numbers carries a high risk of information loss by randomly discarding important variations in the background factors held by the majority, so I clearly judged it inappropriate.
2. Extracting the true important variables contributing to prediction
   • Adopted method: the standardized Boruta algorithm and correlation analysis using the phi coefficient
     To carefully select only the variables truly useful for prediction from among the several hundred created dynamic features, I adopted the “Boruta algorithm.” This method creates “fake variables (shadow features)” by randomly shuffling the order of the original variables, and objectively and rigorously eliminates, within the machine-learning model, “variables that contributed less to prediction than the fake variables.” Furthermore, I performed correlation analysis using the phi coefficient among the variables that survived Boruta, and stabilized the final predictive model by organizing and integrating “variables with similar meaning (a cause of multicollinearity)” whose coefficient exceeded 0.70.
   • Method not adopted: variable selection by univariate-analysis p-values
     The method of evaluating in isolation whether each variable is associated with the response variable (such as the t-test or chi-squared test) completely overlooks the “interactions” whereby multiple variables exert powerful predictive power only in combination, so I refrained from adopting it as a variable-selection method for analyzing complex biological data like this case.
3. Building the machine-learning model and rigorous accuracy evaluation
   • Adopted method: random forest using Bayesian optimization and 10-fold cross-validation (K-Fold Cross Validation)
     I adopted the “random forest,” which builds many decision trees and reaches a conclusion by majority vote. For the complex settings that govern the model’s performance (the number of decision trees, their depth, the minimum number of samples needed for splitting, etc.), rather than relying on rules of thumb, I used “Bayesian optimization estimation” to mathematically search for the best combination. Furthermore, to prove that the built model’s performance was not “just happening to work well with that data split,” I performed “10-fold cross-validation,” in which the data are divided into 10 groups and training and testing are repeated, rigorously evaluating the generalization performance against unseen data.
   • Methods not adopted: ordinary logistic regression analysis or a single decision-tree analysis
     Ordinary logistic regression presupposes that variables have a linear influence, so it cannot appropriately express non-linear thresholds or complex conditional branching such as “the risk jumps only when one symptom worsens and another also overlaps.” Also, a single decision-tree analysis is prone to overfitting and tends to give unstable results, so I judged it insufficient as a predictive model to support clinical decision-making and refrained from adopting it.

Overview of the main results and clinical considerations

As a result of the rigorous accuracy evaluation using cross-validation, it was confirmed that setting the SMOTE-NC over-sampling multiplier excessively high (for example, 10-fold) improved the apparent accuracy on the training data but showed signs of “overfitting,” in which the predictive accuracy on unseen test data declined. As a result of analyzing the variation in the indicators in detail, the “8-fold amplification” model—which maximizes the capture rate of hospitalized patients while suppressing overfitting—was identified as the most generalizable optimal model.

The optimized random-forest model showed an extremely excellent discriminative ability of 0.91 on the evaluation index AUC (area under the ROC curve). The clinically most important “recall (the proportion of patients who actually went to emergency hospitalization who were correctly warned in advance as being at risk of hospitalization)” reached 0.76, confirming that it greatly prevents the oversight of dangerous patients. At the same time, the “precision (the proportion of patients for whom a warning was issued who actually went to emergency hospitalization)” was 0.46, proving that, compared with conventional simple-threshold alerts (whose precision often remains below 10%), it can greatly reduce the “excessive false alarms” that exhaust healthcare workers.

Furthermore, as a result of analyzing the variable importance (an index showing how much each contributed to the model’s prediction) extracted by the Boruta algorithm, it was quantitatively clarified that, more than a single day’s “worsening of the shortness-of-breath score,” the “trajectory pattern in which the shortness-of-breath score worsens in stages for three consecutive days” and the “pattern in which the worsening of lower-limb edema and the decline in quality of sleep occur in the same period” had overwhelmingly higher importance in predicting emergency hospitalization (for example, the presence of a particular continuous-worsening pattern was evaluated as having an importance score 2.61 times higher than other variables).

Dr.DataScience’s contribution

In this case, Dr.DataScience created a predictive model at a level usable in the field by applying the latest machine-learning approaches to the fated challenge of clinical data—”overwhelming data imbalance” that general statistical methods cannot handle.

1. Presenting the best solution to complex data imbalance
   • To predict an extremely rare yet serious clinical event, I implemented advanced sampling techniques combining SMOTE-NC and ENN. This avoided the traps of overfitting and information loss that simple methods tend to fall into, and built a robust data foundation on which the algorithm can learn the true danger signals.
2. Extracting the true risk factors based on an objective algorithm
   • By eliminating variable selection that relies on human intuition or univariate p-values and using the Boruta algorithm, I purified only the indicators with true predictive value from among a vast number of change patterns. This provided new evidence-based guidance for nursing assessment regarding “which symptom trajectories one should focus on when observing patients.”
3. Model evaluation and optimization aligned with clinical-field operations
   • Rather than merely pursuing a high “accuracy rate,” I performed hyperparameter tuning by Bayesian optimization with a deep understanding of the trade-off between “the danger of oversight (recall)” and “exhaustion due to false alarms (precision)” in the medical setting. This provided the client with the specifications of a predictive algorithm with extremely high practicality and safety and a good balance, suitable for incorporation into a remote-monitoring system.