External validation of prognostic scores for COVID-19: a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals

Yannis Lombardi, L. Azoyan, Piotr Szychowiak, Ali Bellamine, Guillaume Lemaître, Mélodie Bernaux, Christel Daniel, Judith Leblanc, Quentin Riller, Olivier Steichen, Pierre‐Yves Ancel, Alain Bauchet, Nathanaël Beeker, Vincent Benoît, Romain Bey, Aurélie Bourmaud, Stéphane Bréant, Anita Burgun, Fabrice Carrat, Charlotte Caucheteux, Julien Champ, Sylvie Cormont, Julien Dubiel, Catherine Duclos, Loïc Estève, Marie Frank, Nicolas Garcelon, Alexandre Gramfort, Nicolas Griffon, Olivier Grisel, Martin Guilbaud, Claire Hassen‐Khodja, F. Hémery, Martin Hilka, Anne‐Sophie Jannot, Jérôme Lambert, Richard Layese, Léo Lebouter, Damien Leprovost, Ivan Lerner, Kankoé Lévi Sallah, Aurélien Maire, Marie‐France Mamzer, Patricia Martel, Arthur Mensch, Thomas Moreau, Antoine Neuraz, Nina Orlova, Nicolás Paris, Bastien Rance, Hélène Ravera, Antoine Rozes, Pierre Rufat, Elisa Salamanca, Arnaud Sandrin, Patricia Serre, Xavier Tannier, Jean‐Marc Tréluyer, Damien Van Gysel, Gaël Varoquaux, Jill-Jênn Vie, Maxime Wack, Perceval Wajsbürt, Demián Wassermann, Éric ZapletalIntensive 2021 · DOI

Abstract

The COVID-19 pandemic led to a surge in prognostic models designed to assist clinical decision-making. However, many lacked robust external validation. This study evaluates 32 high-quality prognostic scores using a large multicenter cohort of 14,343 patients from the Greater Paris University Hospitals. We found that while many scores underperformed compared to their original reports, the 4C Mortality Score and the ABCS score remained highly accurate and robust for predicting 30-day in-hospital mortality.

Introduction

Early in the COVID-19 pandemic, a "tsunami" of prognostic models emerged to help clinicians triage patients and manage limited resources. Most were developed using small, single-center datasets, raising concerns about their generalizability. Variations in healthcare systems, patient demographics, and evolving treatment protocols (e.g., corticosteroid use) can significantly impact a model's performance. This study aims to identify which scores are truly reliable in a real-world clinical setting.

Methods

We conducted a retrospective cohort study using the Greater Paris University Hospitals (GPUH) Clinical Data Warehouse, aggregating data from 39 hospitals.

Score Selection and Evaluation

Using "COVID-19 Evidence Alerts" from McMaster University, we identified 32 "high-quality" scores computable with routine hospital data. To address missing clinical data, we employed multiple imputation (50 datasets). Performance was evaluated using the Area Under the Receiver Operating Characteristic curve (AUC) for 30-day in-hospital mortality and a composite outcome of mortality or ICU transfer.

Figure 1: Flowchart illustrating the patient selection process for the validation cohort.

Results

Dataset Characteristics

The validation cohort included 14,343 consecutive adult patients hospitalized for COVID-19 between January 2020 and April 2021. The primary outcome, 30-day in-hospital mortality, occurred in 2,583 (18%) patients.

Figure 2: Baseline characteristics of the study population, stratified by 30-day mortality status.

Performance of Prognostic Scores

A significant "optimism gap" was observed: 19 of 27 scores performed worse in our cohort than in their original publications. Scores were generally more effective at predicting death than ICU transfer, likely because ICU admission criteria vary by hospital and resource availability.

The top-performing scores for mortality prediction (AUC > 0.75) were:

  • 4C Mortality Score: AUC 0.793
  • ABCS Score: AUC 0.790

Age was the most influential predictor in five of the seven top-performing scores. While discrimination (ranking patients by risk) was strong for top models, many tended to overestimate the absolute risk of death, particularly in later pandemic waves.

Discussion

The resilience of the 4C and ABCS scores across different pandemic waves suggests they capture fundamental pathological aspects of COVID-19. However, the poor performance in predicting ICU transfers highlights the difficulty of modeling outcomes influenced by administrative and clinical judgment rather than purely biological factors.

Limitations

The retrospective design relies on electronic health records, which may have inherent biases. Furthermore, while these scores demonstrate statistical accuracy, their actual clinical utility in improving patient outcomes remains to be evaluated in real-time settings.

Conclusion

The 4C Mortality Score and the ABCS score are the most robust tools for predicting 30-day mortality in hospitalized COVID-19 patients. Clinicians should be cautious with other models that may have been overfitted to specific early-pandemic populations.

Summary updated

BibTeX
@inproceedings{lombardi2021external,
  author = {Yannis Lombardi and L. Azoyan and Piotr Szychowiak and Ali Bellamine and Guillaume Lemaître and Mélodie Bernaux and Christel Daniel and Judith Leblanc and Quentin Riller and Olivier Steichen and Pierre‐Yves Ancel and Alain Bauchet and Nathanaël Beeker and Vincent Benoît and Romain Bey and Aurélie Bourmaud and Stéphane Bréant and Anita Burgun and Fabrice Carrat and Charlotte Caucheteux and Julien Champ and Sylvie Cormont and Julien Dubiel and Catherine Duclos and Loïc Estève and Marie Frank and Nicolas Garcelon and Alexandre Gramfort and Nicolas Griffon and Olivier Grisel and Martin Guilbaud and Claire Hassen‐Khodja and F. Hémery and Martin Hilka and Anne‐Sophie Jannot and Jérôme Lambert and Richard Layese and Léo Lebouter and Damien Leprovost and Ivan Lerner and Kankoé Lévi Sallah and Aurélien Maire and Marie‐France Mamzer and Patricia Martel and Arthur Mensch and Thomas Moreau and Antoine Neuraz and Nina Orlova and Nicolás Paris and Bastien Rance and Hélène Ravera and Antoine Rozes and Pierre Rufat and Elisa Salamanca and Arnaud Sandrin and Patricia Serre and Xavier Tannier and Jean‐Marc Tréluyer and Damien Van Gysel and Gaël Varoquaux and Jill-Jênn Vie and Maxime Wack and Perceval Wajsbürt and Demián Wassermann and Éric Zapletal},
  title = {{External validation of prognostic scores for COVID-19: a multicenter cohort study of patients hospitalized in Greater Paris University Hospitals}},
  booktitle = {Intensive},
  year = {2021},
  doi = {10.1007/s00134-021-06524-w},
}
Figure