TL;DR
This paper demonstrates that a machine learning model trained on non-COVID-19 viral pneumonia patients can effectively predict lethal outcomes in critically ill COVID-19 patients. This "knowledge transfer" approach offers a crucial solution for rapidly developing prognostic tools during the early stages of novel pandemics, outperforming existing clinical scores and even some COVID-19-specific models.
What Problem Does This Paper Address?
The emergence of a novel infectious disease, like COVID-19, presents a critical challenge for healthcare systems: the lack of established, disease-specific prognostic models. In the early phase of a pandemic, predicting which critically ill patients are most likely to succumb is vital for effective resource allocation, treatment prioritization, and clinical decision-making. Developing these models from scratch requires time and extensive data, which are scarce when a new disease first appears. This paper addresses this gap by investigating whether models trained on data from a similar, well-understood disease (non-COVID-19 viral pneumonia) can be successfully applied to predict outcomes in the novel disease (COVID-19).
Methods
The core of this study involves training and testing machine learning models. Specifically, the researchers employed gradient boosted decision tree models, a powerful ensemble learning technique known for its accuracy and ability to handle complex, non-linear relationships in data.
For model development, they first trained a model on a dataset of critically ill patients with non-COVID-19 viral pneumonia. This model was designed to predict the probability of ICU mortality within the next 5 days. To achieve this, they aggregated patient data into 24-hour time bins and assigned a positive class label to bins occurring within 5 days of death.
To understand which features were most influential, they performed feature selection using SHAP (SHapley Additive exPlanations) values. This technique quantifies the contribution of each feature to the model's prediction, allowing them to identify the top 20 most important clinical features. A "reduced model" was then trained using only these top features.
Finally, both the full and reduced models were applied to a separate dataset of critically ill COVID-19 patients to evaluate their predictive performance. This performance was then compared against established clinical scores (APACHE2, SAPS2, SOFA) and previously published COVID-19 mortality prediction models.
Datasets
The study utilized two primary datasets:
- Non-COVID-19 Viral Pneumonia Dataset: This dataset comprised data from 718 critically ill patients with confirmed non-COVID-19 viral pneumonia. This served as the training dataset for developing the machine learning model.
- COVID-19 Dataset: This dataset included data from 1054 critically ill patients with confirmed COVID-19. This dataset was used for testing and evaluating the model trained on the non-COVID-19 data.
The paper does not introduce a new benchmark dataset but rather leverages existing clinical data from two distinct patient cohorts.
Key Insights and Ideas
- "Knowledge Transfer" is Viable: The most significant insight is that predictive models for lethal outcomes can be effectively transferred from one viral pneumonia to another, even when the diseases are novel and differ in their course and severity. This suggests that the underlying physiological mechanisms of organ failure in critical illness share commonalities across different viral pneumonias.
- Early Pandemic Preparedness: This approach offers a practical strategy for rapidly developing prognostic tools during the initial, critical phase of a new pandemic, bridging the gap until disease-specific models can be built.
- Machine Learning Outperforms Traditional Scores: The developed machine learning model significantly outperformed established clinical scores like APACHE2, SAPS2, and SOFA, highlighting the power of ML in complex medical prediction tasks.
- Feature Importance Reveals Common Pathways: The fact that the top predictive features identified in the non-COVID-19 training set were also highly important for predicting COVID-19 mortality suggests that common pathophysiological pathways are at play in severe viral pneumonia. Features related to organ distress and failure (e.g., thrombocytes, heart rate, blood pressure, oxygen saturation) were particularly prominent.
- Reduced Model Sufficiency: A model using only the top 20 features achieved performance comparable to the full model, indicating that a parsimonious set of clinical indicators can be highly effective.
Key Results
The machine learning model trained on non-COVID-19 viral pneumonia patients demonstrated strong predictive performance when applied to COVID-19 patients:
- AUROC: The model achieved an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.86 (95% CI 0.86-0.87).
- Superiority over Clinical Scores: This performance was significantly better than established clinical scores: APACHE2 (0.63), SAPS2 (0.72), and SOFA (0.76).
- Competitive with COVID-19 Specific Models: The model also outperformed some existing COVID-19 specific models, such as Wang's laboratory model (0.62) and clinical model (0.56), and was comparable to Zhou's model (0.76) and the 4C COVID-19 Mortality score (0.71).
- Time-Varying Performance: Predictive metrics generally increased with time after admission and peaked closer to the endpoint (death), suggesting the model becomes more accurate as the patient's condition progresses.
Limitations and Future Work
- Retrospective Data: The study relied on retrospective data, meaning the analysis was limited to parameters that were already recorded in electronic health records. This excluded potentially valuable laboratory markers (e.g., specific cytokine levels) that might have further improved predictive accuracy.
- Comparability Issues: The exclusion of certain patients from the comparison with previously published models (due to data availability) limited direct comparability.
- Generalizability: While the study demonstrates feasibility, further research is needed to investigate how these models can be practically integrated into clinical workflows, particularly in resource-constrained pandemic settings, to assist in the ethical allocation of intensive care resources.
- Future Pandemics: The authors propose that for future pandemics, readily available prediction models from similar diseases should be considered for immediate application to support critical care management and resource allocation.
Summary updated