Abstract
We investigate machine learning models that identify COVID-19 positive patients and estimate the mortality risk based on routinely acquired blood tests in a hospital setting. However, during pandemics or new outbreaks, disease and testing characteristics change, thus we face domain shifts. Domain shifts can be caused, e.g., by changes in the disease prevalence (spreading or tested population), by refined RT-PCR testing procedures (taking samples, laboratory), or by virus mutations. Therefore, machine learning models for diagnosing COVID-19 or other diseases may not be reliable and degrade in performance over time. To countermand this effect, we propose methods that first identify domain shifts and then reverse their negative effects on the model performance. Frequent re-training and reassessment, as well as stronger weighting of more recent samples, keeps model performance and credibility at a high level over time. Our diagnosis models are constructed and tested on large-scale data sets, steadily adapt to observed domain shifts, and maintain high ROC AUC values along pandemics.
Introduction: The Challenge of Domain Shifts in Pandemics
The COVID-19 pandemic highlighted the need for rapid, scalable diagnostic tools. While RT-PCR is the gold standard, its cost and turnaround time limit its utility for mass screening. Routine blood tests, which are inexpensive and ubiquitous, offer a promising alternative when paired with machine learning (ML). However, a significant hurdle in clinical ML is domain shift—the phenomenon where changes in disease prevalence, virus variants, or hospital protocols cause model performance to degrade over time. Standard ML models often assume static data distributions, leading to "unreliable performance estimates" in dynamic environments like a pandemic.
Methods: Lifelong Learning and Temporal Weighting
To address domain shifts, this study proposes a "lifelong learning" framework. Instead of a static model, the authors advocate for frequent re-training (e.g., monthly) to incorporate the most recent clinical data.
- Temporal Weighting: A weighting scheme was implemented where recent samples are prioritized (e.g., weighted 3x higher than older data), allowing the model to adapt to the current "domain" of the pandemic.
- Prospective Evaluation: To simulate real-world deployment, models were trained on past data and tested on subsequent months, avoiding the "random shuffle trap" that often leads to over-optimistic results.
Datasets and Cohort Selection
The research utilized a large-scale dataset from Kepler University Hospital in Linz, Austria, comprising over 127,000 samples. The cohort included pre-pandemic negatives to establish a baseline for non-COVID illnesses and pandemic-era samples confirmed via RT-PCR.
Key Results and Insights
- Diagnosis Performance: In prospective evaluations (testing on Nov/Dec 2020), the Extreme Gradient Boosting (XGB) model achieved a ROC AUC of 0.81. This is significantly more realistic than the ~0.97 ROC AUC obtained through standard random cross-validation, which fails to account for temporal shifts.
- Mortality Prediction: The models proved highly effective at identifying high-risk patients, with a Random Forest achieving a ROC AUC of 0.86 in temporal evaluation.
- Biomarkers: Key diagnostic indicators included Ferritin, Fibrinogen, and Calcium. For mortality, Neutrophils and LDH were the strongest predictors.
Discussion and Future Work
A primary finding is that Precision-Recall (PR) AUC is highly sensitive to disease prevalence. As the pandemic fluctuates, a model's utility can change, necessitating constant monitoring. The main limitation remains inter-hospital domain shift; models trained at one facility may require re-calibration before deployment elsewhere due to differences in lab equipment and patient demographics.
By implementing lifelong learning and temporal weighting, ML models can maintain high diagnostic accuracy despite the evolving nature of a pandemic. This approach transforms routine blood tests into a dynamic, robust tool for pandemic management.
Summary updated