Abstract
Many previous studies claim to have developed machine learning models that diagnose COVID-19 from blood tests. However, we hypothesize that changes in the underlying distribution of the data, so called domain shifts, affect the predictive performance and reliability and are a reason for the failure of such machine learning models in clinical application. Domain shifts can be caused, e.g., by changes in the disease prevalence, refined RT-PCR testing procedures, or virus mutations. We reveal domain shifts by evaluating models on a large-scale dataset with different assessment strategies, such as temporal validation. We present the novel finding that domain shifts strongly affect machine learning models for COVID-19 diagnosis and deteriorate their predictive performance and credibility. Therefore, frequent re-training and re-assessment are indispensable for robust models enabling clinical utility.
Introduction: The Problem of Static Models in a Dynamic Pandemic
In the early stages of the COVID-19 pandemic, numerous machine learning (ML) models were proposed to diagnose the virus using routine blood tests. Despite promising initial results, these models rarely achieved successful clinical implementation. This study identifies domain shift as the primary obstacle. In a medical context, the "ground truth" is not static; factors such as the emergence of new variants, changing hospital admission criteria, and seasonal fluctuations in blood chemistry alter the data distribution over time.
Most existing studies rely on "random cross-validation," which shuffles all available data. This approach inadvertently allows models to "peek" into the future, leading to an overestimation of real-world reliability. This research investigates how these shifts affect model performance and credibility.
Methodology
The researchers benchmarked several ML architectures, including Self-Normalizing Neural Networks (SNN), K-Nearest Neighbors (KNN), Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and Extreme Gradient Boosting (XGB).
The core of the methodology lies in the comparison of two validation strategies:
- Random Validation: The standard approach of shuffling and splitting data.
- Temporal Validation: Training the model on data from one period (e.g., March to October) and testing it on a subsequent period (November to December).
Feature engineering involved the 100 most frequent blood test parameters. Notably, the researchers included binary "missingness" flags to indicate whether a specific test was ordered, as clinical suspicion itself carries significant diagnostic information.
Dataset Composition
The study utilized a massive dataset from the Kepler University Hospital in Linz, Austria, comprising nearly 80,000 cases. This scale significantly exceeds most previous small-scale COVID-19 ML studies.
The data was structured into two primary cohorts:
- 2019 Cohort: Approximately 70,000 pre-pandemic cases used as guaranteed "true negatives" to establish a baseline for healthy and non-COVID illness distributions.
- 2020 Cohort: Approximately 9,000 cases from the pandemic period, including 1,037 COVID-positive cases confirmed via RT-PCR.
Key Insights and Results
The study revealed a significant "overoptimism trap." While standard random validation yielded ROC AUC scores as high as 0.96, prospective temporal validation saw performance drop to 0.81. This suggests that many published models over-report their clinical utility by failing to account for temporal shifts.
Tree-based ensemble methods, specifically XGBoost and Random Forest, consistently outperformed neural networks and linear models. For COVID-19 diagnosis, the XGB model achieved an ROC AUC of 0.916 in random validation but fell to 0.814 in temporal validation. Mortality prediction followed a similar trend, with Random Forest dropping from 0.881 to 0.857.
The authors also observed that blood chemistry is a "moving target." Visualizations of parameters like Chloride and Erythrocytes showed fluctuations throughout 2020, representing fundamental shifts in patient populations and the virus's physiological impact.
Conclusion and Clinical Implications
Domain shifts actively erode model credibility over time. To maintain clinical utility, ML models must be treated as living systems. The authors recommend frequent re-training on the most recent data and potentially weighting recent samples more heavily to help models adapt to emerging shifts. Future work should test these findings across different geographic regions and healthcare systems to develop more robust, universal diagnostic tools.
Summary updated