Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2

Honghan Wu, Huayu Zhang, Andreas Karwath, Zina M. Ibrahim, Ting Shi, Xin Zhang, Kun Wang, Jiaxing Sun, Kevin Dhaliwal, Daniel Bean, Victor Roth Cardoso, Kezhi Li, James T. Teo, Amitava Banerjee, Fang Gao-Smith, Tony Whitehouse, Tonny Veenith, Georgios V. Gkoutos, Xiaodong Wu, Richard J. B. Dobson, Bruce GuthrieJ. Am. Medical Informatics Assoc 2021 · DOI

TL;DR

This paper tackles the challenge of creating robust COVID-19 prognosis prediction models that perform consistently across diverse patient populations. They developed a novel ensemble learning framework that synergizes existing models, demonstrating superior performance and calibration compared to individual models across four international cohorts. This approach offers a promising path towards more reliable clinical decision support for infectious diseases.

What Problem Does This Paper Address?

Accurate risk prediction models are crucial for effective clinical decision-making, especially during public health crises like the COVID-19 pandemic. However, models developed from single cohorts often struggle to generalize to diverse populations due to variations in demographics, disease severity, and healthcare system responses. This leads to inconsistent performance, limiting their utility at a population level. The authors aim to overcome this by creating a more robust and adaptable prediction system by combining multiple existing models.

Methods

The core of this work is a novel ensemble learning framework designed to synergize pre-existing COVID-19 prognosis models. Instead of training new models from scratch, they selected and reimplemented seven published models derived from diverse cohorts and using various techniques. Their ensemble approach involves a "competence assessment framework" to quantify how well each individual model performs for a given patient. This framework considers three factors:

  • Familiarity competence: How similar the patient's characteristics are to the derivation cohort of a given model.
  • General competence: Based on the size of the model's derivation cohort (larger cohorts generally lead to more robust models).
  • Data completeness: Whether the patient's data includes all the predictors required by a specific model.

This competence score is then used in a "wisdom-of-the-crowd" approach (weighted averaging of predictions) to generate the final ensemble prediction. This method is data-independent, meaning it can be applied to new datasets without requiring further local training.

Datasets

The study utilized four diverse international cohorts for validation:

  • Wuhan01 Cohort (China): 2869 adults with COVID-19.
  • Wuhan02 Cohort (China): 357 adults with COVID-19.
  • King's College Hospital (KCH) Cohort (UK): 1475 adults hospitalized with COVID-19.
  • University Hospitals Birmingham (UHB) Cohort (UK): 693 adults hospitalized with COVID-19.

These cohorts differ in size, mortality rates, and importantly, in their admission strategies, providing a robust testbed for model generalizability. The authors also referenced the derivation cohorts of the seven individual models, which were from various regions in China and the UK.

Key Insights and Ideas

  • No single model is a silver bullet: The study clearly demonstrates that individual prediction models, even those with good performance on their original datasets, often fail to generalize consistently across different populations. This highlights the inherent challenge of developing universally applicable predictive tools.
  • Complementarity is key: The success of the ensemble model suggests that individual models, when derived from diverse and complementary cohorts, can collectively capture a broader spectrum of patient presentations and outcomes. The ensemble leverages the strengths of each model for specific patient profiles.
  • Context matters for prediction: The significant performance disparity between Chinese and UK cohorts underscores the impact of healthcare system responses (e.g., admission strategies) on disease prognosis and the predictive power of certain features. This implies that models might need to be adapted or weighted differently based on the clinical context.
  • Early data is crucial: The authors hypothesize that blood parameters and physiological measurements collected early in the disease course might be more predictive, especially in scenarios where healthcare systems are strained.

Key Results

  • Superior Discrimination: The ensemble model consistently achieved the best discrimination (measured by C-index) across all four validation cohorts, outperforming all individual models. For instance, it achieved C-indices of 0.914 (Wuhan01), 0.890 (Wuhan02), 0.665 (KCH), and 0.683 (UHB).
  • Robust Calibration: The ensemble model also demonstrated consistently good to very good calibration across all cohorts, meaning its predicted probabilities closely matched observed outcomes.
  • Improved Clinical Usefulness: The ensemble model outperformed individual models across metrics like positive predictive value, sensitivity, and specificity, indicating better utility for clinical decision-making.
  • Performance Disparities: All models, including the ensemble, performed better on the Chinese cohorts compared to the UK cohorts, likely due to differences in admission strategies and the timing of data collection relative to disease progression.
  • Net Reclassification Improvements: The ensemble model showed significant net reclassification improvements compared to the best individual model in each cohort, particularly on the Wuhan02 cohort (4.83%).

Limitations and Future Work

  • Limited European Cohorts: The study acknowledges a limitation in not being able to include prediction models derived from European cohorts, particularly from the UK. Including more local models could further improve the ensemble's performance in UK cohorts.
  • Future Platform: The authors propose creating a web platform to allow the community to share models, enabling the synergy of a wider range of diverse and complementary models. This would facilitate more robust ensemble predictions.
  • Data Availability: While the code and metadata are available, the Wuhan01 and Wuhan02 datasets are not publicly available due to anonymization challenges. Access to KCH and UHB datasets is available upon request under specific conditions.

Summary updated

BibTeX
@article{wu2021ensemble,
  author = {Honghan Wu and Huayu Zhang and Andreas Karwath and Zina M. Ibrahim and Ting Shi and Xin Zhang and Kun Wang and Jiaxing Sun and Kevin Dhaliwal and Daniel Bean and Victor Roth Cardoso and Kezhi Li and James T. Teo and Amitava Banerjee and Fang Gao-Smith and Tony Whitehouse and Tonny Veenith and Georgios V. Gkoutos and Xiaodong Wu and Richard J. B. Dobson and Bruce Guthrie},
  title = {{Ensemble learning for poor prognosis predictions: A case study on SARS-CoV-2}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2021},
  doi = {10.1093/jamia/ocaa295},
}
Figure