Generalisability of AI-based scoring systems in the ICU: a systematic review and meta-analysis

Patrick Rockenschaub, Ela M. Akay, Benjamin Gregory Carlisle, Adam Hilbert, Falk Meyer-Eschenbach, Anatol‐Fiete Näher, Dietmar Frey, Vince I. MadaimedRxiv 2023 · DOI

TL;DR

While AI-based scoring systems for the ICU are proliferating, only about 11% are ever tested on data from a different hospital. When they are, performance typically drops—sometimes by more than 50%—and the research community is currently over-reliant on just two major US-based open-source datasets, potentially masking a significant "translational gap."

What Problem Does This Paper Address?

The "translational gap" in medical AI is the distance between a model performing well in a research paper and that same model actually helping patients in a clinic. In the ICU, machine learning (ML) models are designed to provide early warnings for acute deterioration (like sepsis or kidney failure). However, these models often "overfit" to the specific quirks of the hospital where they were trained—coding practices, patient demographics, or local clinical protocols.

Without external validation—testing the model on data from a geographically distinct hospital—we have no way of knowing if a model is truly robust or just a "local hero." This paper systematically reviews the literature to see how often researchers actually perform this critical check and how much performance degrades when they do.

Methods

The authors conducted a systematic review and meta-analysis following PRISMA guidelines. They screened over 3,800 records from MEDLINE, Web of Science, and arXiv, eventually including 355 studies that used ML to predict acute ICU deterioration from routine electronic health record (EHR) data.

To quantify the "generalisation gap," they used Bayesian linear mixed-effects models to analyze the change in the Area Under the Receiver Operating Characteristic (AUROC) between internal and external validation. This approach allowed them to account for sampling variability and the hierarchical nature of the data (multiple models within a single study).

Datasets

The study highlights a massive concentration in the data sources used for ICU AI research:

  • MIMIC (Medical Information Mart for Intensive Care): Used in 74.4% of validated studies.
  • eICU Collaborative Research Database: Used in 43.6% of validated studies.
  • The "Big Two" Dominance: Together, MIMIC and eICU accounted for 79.5% of all externally validated studies.
  • Others: Newer public datasets like AUMCdb (Netherlands) and HiRID (Switzerland) appeared in only about 5% of the included studies.

Key Insights and Ideas

  • The Validation Desert: Despite the hype around medical AI, external validation remains the exception, not the rule. Only 11% of the 355 studies performed it, though this is trending upward (reaching ~18% by 2022).
  • The Open-Data Paradox: Open-source datasets like MIMIC are essential for research, but they've created a "monoculture." If a model is developed on MIMIC and validated on eICU, it’s still only being tested on US-based clinical practices. We may be building models that generalize to "US open-source data" rather than to "global clinical practice."
  • AUROC is a Blunt Instrument: The authors suggest that the standard metric, AUROC, might be too optimistic. In some cases, a "mild" drop in AUROC actually corresponded to a model becoming "worse than useless" when measured by clinical utility metrics (like the ability to predict sepsis within a specific actionable window).
  • The "Easier Patient" Effect: Interestingly, in 9.1% of cases, performance actually increased during external validation. This likely points to "case-mix" differences—the validation hospital might have had "easier" patients to predict, which can misleadingly suggest a model is more robust than it actually is.

Key Results

  • Performance Drop: On average, the AUROC decreased by -0.037 (95% CI -0.064 to -0.017) when moving from internal to external data.
  • Significant Failures: In 38.6% of studies, the performance loss was greater than 0.05 AUROC. In extreme cases, performance dropped by more than 50% relative to the training cohort.
  • Mortality vs. Others: Models predicting mortality were validated less frequently (6.5%) than models predicting other outcomes like sepsis or kidney injury.
  • Trend: Over 80% of all identified external validations were published in 2018 or later, showing a growing (but still insufficient) awareness of the generalisability problem.

Limitations and Future Work

  • Best-Case Bias: The authors acknowledge that the 11% of studies that did perform external validation are likely the highest-quality papers. The performance drop among the other 89% of non-validated models is likely much worse.
  • Metric Limitations: The meta-analysis relied almost exclusively on AUROC because other metrics (like AUPRC or clinical utility) are reported too infrequently to aggregate.
  • Risk of Bias: The authors did not perform a formal risk-of-bias assessment for the individual included studies.
  • Future Direction: The paper calls for external validation to become the default requirement for publishing new ICU scoring systems. They also recommend that researchers report performance on a model trained only on the validation data to provide a "ceiling" for comparison.

Summary updated

Abstract

Abstract Background Machine learning (ML) is increasingly used to predict clinical deterioration in intensive care unit (ICU) patients through scoring systems. Although promising, such algorithms often overfit their training cohort and perform worse at new hospitals. Thus, external validation is a critical – but frequently overlooked – step to establish the reliability of predicted risk scores to translate them into clinical practice. We systematically reviewed how regularly external validation of ML-based risk scores is performed and how their performance changed in external data. Methods We searched MEDLINE, Web of Science, and arXiv for studies using ML to predict deterioration of ICU patients from routine data. We included primary research published in English before April 2022. We summarised how many studies were externally validated, assessing differences over time, by outcome, and by data source. For validated studies, we evaluated the change in area under the receiver operating characteristic (AUROC) attributable to external validation using linear mixed-effects models. Results We included 355 studies, of which 39 (11.0%) were externally validated, increasing to 17.9% by 2022. Validated studies made disproportionate use of open-source data, with two well-known US datasets (MIMIC and eICU) accounting for 79.5% of studies. On average, AUROC was reduced by -0.037 (95% CI -0.064 to -0.017) in external data, with >0.05 reduction in 38.6% of studies. Discussion External validation, although increasing, remains uncommon. Performance was generally lower in external data, questioning the reliability of some recently proposed ML-based scores. Interpretation of the results was challenged by an overreliance on the same few datasets, implicit differences in case mix, and exclusive use of AUROC.
BibTeX
@misc{rockenschaub2023generalisability,
  author = {Patrick Rockenschaub and Ela M. Akay and Benjamin Gregory Carlisle and Adam Hilbert and Falk Meyer-Eschenbach and Anatol‐Fiete Näher and Dietmar Frey and Vince I. Madai},
  title = {{Generalisability of AI-based scoring systems in the ICU: a systematic review and meta-analysis}},
  year = {2023},
  doi = {10.1101/2023.10.11.23296733},
  howpublished = {medRxiv},
}
Figure