Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective cohort study

Jim M Smit, Jesse H. Krijthe, Andrei N. Tintu, Henrik Endeman, Jeroen Ludikhuize, Michel E. van Genderen, Shermarke Hassan, Rachida el Moussaoui, Peter E. Westerweel, Robbert J Goekoop, Geeke Waverijn, Tim Verheijen, Jan G. den Hollander, Mark de Boer, Diederik Gommers, Robin van der Vlies, Mark W.m. Schellings, Regina A. Carels, Cees van Nieuwkoop, M. Sesmu Arbous, Jasper van Bommel, Rachel Knevel, Yolanda B. de Rijke, Marcel J. T. ReindersIntensive 2022 · DOI

Abstract

This study demonstrates the potential benefit of transitioning from general early warning models to models specifically tailored for hospitalized COVID-19 patients. By employing machine learning and dynamic updating, the researchers developed a model that significantly outperforms standard clinical scores. While promising, further independent validation is required before widespread clinical implementation.

Introduction

In hospital settings, clinicians rely on Early Warning Scores (EWS) to identify patients at risk of clinical deterioration. However, standard tools like the National Early Warning Score (NEWS) were designed for general patient populations and often underestimate the severity of COVID-19.

COVID-19 presents unique clinical challenges, such as "silent hypoxia," where patients exhibit dangerously low oxygen levels without obvious respiratory distress. Furthermore, the pandemic was a rapidly evolving crisis; treatment protocols (such as the introduction of dexamethasone) and virus variants changed frequently. This study addresses the need for a longitudinal monitoring tool that is both COVID-specific and capable of adapting to a changing clinical landscape.

Methods and Study Design

The researchers developed a Random Forest (RF) model to predict clinical deterioration—defined as ICU admission or unexpected death—within a 24-hour window. The study utilized a multi-center retrospective cohort consisting of 3,514 COVID-19 patient admissions across six Dutch hospitals (two academic and four teaching hospitals) between February 2020 and May 2021.

Overview of the study timeline, dynamic model updating workflow (a), and patient inclusion flowchart (b)

Key methodological highlights include:

  • Feature Engineering: Beyond standard vitals, the authors introduced the $SpO_2/O_2$ ratio to quantify oxygenation efficiency and included "delta" ($\Delta$) values to capture the trend of vital signs over time.
  • Dynamic Model Updating: To account for changing treatments and variants, the team proposed a monthly updating strategy. Each month, the model was retrained using the most recent data from other hospitals and recalibrated using local hospital data to ensure continued accuracy.
  • Explainability: The study employed SHAP (SHapley Additive exPlanations) values to provide transparency, allowing clinicians to see which specific factors were driving a patient's risk score.

Results

The study cohort's characteristics varied significantly based on clinical outcomes, as detailed in the table below. Patients who required ICU admission or died generally exhibited higher respiratory rates and lower $SpO_2/O_2$ ratios compared to those discharged alive.

Baseline clinical characteristics and vital signs of the study cohort, categorized by outcome

Model Performance

  • Superior Discrimination: The dynamic RF model achieved a partial Area Under the Curve (pAUC) of 0.82 [0.80–0.84], significantly higher than the National Early Warning Score (0.72) and the Modified Early Warning Score (0.67).
  • Top Predictors: SHAP analysis identified the $SpO_2/O_2$ ratio, respiratory rate, temperature, and ward length-of-stay as the most critical factors for predicting deterioration. The $SpO_2/O_2$ ratio emerged as the single most important predictor, capturing the intensity of oxygen support required—a more sensitive indicator of lung failure than binary "yes/no" oxygen therapy markers.
  • Clinical Utility: Decision Curve Analysis (DCA) demonstrated that the model provided a higher "net benefit" across all clinically relevant threshold probabilities, effectively catching more deteriorating patients while minimizing the "alert fatigue" caused by unnecessary alarms.

Discussion and Limitations

The dynamic updating strategy proved essential for maintaining model calibration. It ensures that a predicted risk (e.g., 10%) actually corresponds to the observed event rate, even as baseline risks shift due to improved medical interventions.

However, several limitations remain:

  • Retrospective Bias: The model was trained on data where clinicians were already intervening. If a doctor moved a patient to the ICU early based on clinical judgment, the model might have mislabeled preceding data points.
  • Missing Data: The study relied on statistical imputation for missing values, assuming data was missing at random—an assumption that may not hold in high-stress clinical environments.
  • Generalizability: The data is entirely from the Netherlands. Further validation is needed in different healthcare systems and against newer variants like Omicron.

Conclusion

Tailoring early warning models to specific disease states like COVID-19 offers significant performance gains over general-purpose tools. The inclusion of oxygenation ratios and dynamic updating allows these models to remain relevant in a shifting clinical environment. Future work should focus on prospective, interventional studies to prove that implementing this model improves patient outcomes and reduces healthcare worker workload.

Summary updated

Abstract

This study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. Further (independent) validation of the model is needed.
BibTeX
@inproceedings{smit2022development,
  author = {Jim M Smit and Jesse H. Krijthe and Andrei N. Tintu and Henrik Endeman and Jeroen Ludikhuize and Michel E. van Genderen and Shermarke Hassan and Rachida el Moussaoui and Peter E. Westerweel and Robbert J Goekoop and Geeke Waverijn and Tim Verheijen and Jan G. den Hollander and Mark de Boer and Diederik Gommers and Robin van der Vlies and Mark W.M. Schellings and Regina A. Carels and Cees van Nieuwkoop and M. Sesmu Arbous and Jasper van Bommel and Rachel Knevel and Yolanda B. de Rijke and Marcel J. T. Reinders},
  title = {{Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective cohort study}},
  booktitle = {Intensive},
  year = {2022},
  doi = {10.1186/s40635-022-00465-4},
}
Figure