Abstract
The COVID-19 pandemic led to a surge in prognostic models designed to assist clinical decision-making. However, many lacked robust external validation. This study evaluates 32 high-quality prognostic scores using a large multicenter cohort of 14,343 patients from the Greater Paris University Hospitals. We found that while many scores underperformed compared to their original reports, the 4C Mortality Score and the ABCS score remained highly accurate and robust for predicting 30-day in-hospital mortality.
Introduction
Early in the COVID-19 pandemic, a "tsunami" of prognostic models emerged to help clinicians triage patients and manage limited resources. Most were developed using small, single-center datasets, raising concerns about their generalizability. Variations in healthcare systems, patient demographics, and evolving treatment protocols (e.g., corticosteroid use) can significantly impact a model's performance. This study aims to identify which scores are truly reliable in a real-world clinical setting.
Methods
We conducted a retrospective cohort study using the Greater Paris University Hospitals (GPUH) Clinical Data Warehouse, aggregating data from 39 hospitals.
Score Selection and Evaluation
Using "COVID-19 Evidence Alerts" from McMaster University, we identified 32 "high-quality" scores computable with routine hospital data. To address missing clinical data, we employed multiple imputation (50 datasets). Performance was evaluated using the Area Under the Receiver Operating Characteristic curve (AUC) for 30-day in-hospital mortality and a composite outcome of mortality or ICU transfer.
Results
Dataset Characteristics
The validation cohort included 14,343 consecutive adult patients hospitalized for COVID-19 between January 2020 and April 2021. The primary outcome, 30-day in-hospital mortality, occurred in 2,583 (18%) patients.
Performance of Prognostic Scores
A significant "optimism gap" was observed: 19 of 27 scores performed worse in our cohort than in their original publications. Scores were generally more effective at predicting death than ICU transfer, likely because ICU admission criteria vary by hospital and resource availability.
The top-performing scores for mortality prediction (AUC > 0.75) were:
- 4C Mortality Score: AUC 0.793
- ABCS Score: AUC 0.790
Age was the most influential predictor in five of the seven top-performing scores. While discrimination (ranking patients by risk) was strong for top models, many tended to overestimate the absolute risk of death, particularly in later pandemic waves.
Discussion
The resilience of the 4C and ABCS scores across different pandemic waves suggests they capture fundamental pathological aspects of COVID-19. However, the poor performance in predicting ICU transfers highlights the difficulty of modeling outcomes influenced by administrative and clinical judgment rather than purely biological factors.
Limitations
The retrospective design relies on electronic health records, which may have inherent biases. Furthermore, while these scores demonstrate statistical accuracy, their actual clinical utility in improving patient outcomes remains to be evaluated in real-time settings.
Conclusion
The 4C Mortality Score and the ABCS score are the most robust tools for predicting 30-day mortality in hospitalized COVID-19 patients. Clinicians should be cautious with other models that may have been overfitted to specific early-pandemic populations.
Summary updated