An Ensemble approach for Ensemble-Modelled Cardiac Surgery Risk Evaluation, Data Usage and Clinical Interpretability

Gianni D. Angelini, Tim Dong, Shubhra Sinha, Bing Zhai, Daniel Fudulu, Arnaldo Dimagli, Pradeep Narayan, Jeremy Chan, Chris Holmes, Massimo Caputo, Lauren Dixon, Marco Gemelli, Hunaid A. Vohra, Umberto BenedettoResearch Square 2022 · DOI

TL;DR

Researchers have developed a sophisticated "ensemble of ensembles" approach to predict cardiac surgery mortality, successfully combining decades of "siloed" medical data that use different clinical variables. By moving beyond simple accuracy metrics to a composite "Clinical Effectiveness Metric" (CEM), the study demonstrates that machine learning ensembles significantly outperform traditional risk scores and adapt better to changes in clinical practice over time.

What Problem Does This Paper Address?

In cardiac surgery, risk stratification—predicting the likelihood of a patient surviving an operation—is vital for informed consent and clinical decision-making. However, two major hurdles exist:

  1. Calibration Drift: Medical practice evolves. A risk model built in 1999 becomes less accurate by 2019 because surgical techniques, anesthesia, and patient populations change. This "drift" means old models eventually overestimate or underestimate risk.
  2. Data Silos: As clinical standards evolve, the variables we track change. For example, the older "Logistic EuroScore" (LogES) uses one set of variables, while the newer "EuroScore II" (ES II) uses another. This creates "siloed" datasets that are difficult to combine because they don't "speak the same language," leading to a loss of predictive power from historical data.

The authors sought to determine if ensemble machine learning could bridge these eras of data and provide a more stable, interpretable, and clinically useful tool than traditional statistical methods.

Methods

The researchers employed a multi-layered ensemble strategy. They started with six "base learner" algorithms:

  • Logistic Regression
  • Neural Networks
  • Random Forest (RF)
  • Weighted Support Vector Machines (SVM)
  • XGBoost
  • Bayesian Update (a traditional statistical recalibration method)

They trained these models on two different variable sets (LogES and ES II) across different time partitions. To combine them, they used homogeneous ensembles (same algorithm, different data samples) and heterogeneous ensembles (different algorithms combined).

A key innovation was the Clinical Effectiveness Metric (CEM). Rather than relying solely on the Area Under the Curve (AUC)—which can be misleading in medical data with low event rates—the CEM uses a geometric mean of five factors: discrimination (AUC, F1 score), calibration (Expected Calibration Error), overall accuracy (Brier score), and clinical utility (Net Benefit Analysis). Finally, they used SHAP (SHapley Additive exPlanations) to ensure the "black box" models remained interpretable for surgeons.

Datasets

The study utilized the National Adult Cardiac Surgery Audit (NACSA) dataset, a massive registry covering nearly all cardiac surgeries in the UK.

  • Size: 647,726 patients from 45 hospitals.
  • Timeframe: January 1996 to March 2019.
  • Splits: The data was partitioned into Training (1996–2011), Update (2012–2016), and Holdout (2017–2019) cohorts to rigorously test the models' ability to handle temporal shifts.

Key Insights and Ideas

  • Diversity Wins: The "No Free Lunch Theorem" suggests no single model is best for every data distribution. The authors found that combining diverse models (heterogeneous ensembles) helped mitigate the weaknesses of individual algorithms, especially when dealing with the transition between different clinical scoring eras.
  • The Power of Siloed Data: One of the most significant "aha" moments was that combining siloed datasets across time (using different variables) yielded significantly better performance (CEM 0.720) than building models on the most recent data alone. This proves that "old" data still holds valuable patterns if the ensemble architecture is right.
  • Cognitive Efficiency: By condensing multiple performance metrics into a single CEM, the researchers addressed "Miller’s Law" (the limit of human short-term memory). This makes it easier for clinicians to compare and select the best-performing models without getting lost in a sea of statistical tables.

Key Results

  • Top Performer: The XGBoost homogeneous ensemble was the overall winner, achieving a CEM of 0.725 and an AUC of 0.8327.
  • Superiority over Tradition: Both homogeneous and heterogeneous ML ensembles significantly outperformed the traditional Bayesian Update recalibration method.
  • Interpretability: SHAP analysis revealed that the "weight of intervention" (complexity of the surgery), operative urgency, age, and NYHA class (heart failure severity) were the strongest predictors of mortality. Interestingly, it also captured complex interactions, such as the protective effect of dialysis in certain renal impairment scenarios.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Recency: The dataset only included three months of data from 2019.
  • Class Imbalance: Mortality in cardiac surgery is low (~3.3%), which creates a "needle in a haystack" problem for ML models. While the ensemble approach helped, further work on over/under-sampling techniques is suggested.
  • The COVID Factor: The study uses pre-pandemic data. Whether these models remain stable in a post-pandemic world—where patient profiles and hospital resources have shifted—remains an open question.
  • Metric Refinement: Future work could explore optimizing for other specific metrics, like the F1 score, to further improve sensitivity in high-risk cases.

Summary updated

Abstract

<title>Abstract</title> Risk stratification plays a major role in the clinical decision-making process, patient consent and clinical governance analysis. However, the calibration of current risk scores (e.g., European System for Cardiac Operative Risk Evaluation (EuroSCORE), The Society of Thoracic Surgeons (STS) risk score) has been shown to deteriorate over time – a process known as calibration drift. The introduction of new clinical scores with different variable sets typically result in disparate datasets due to different levels of missingness. This is a barrier to the full insight and predictive capability of datasets across all potentially available time ranges. Little is known about the use of ensemble learning with ensemble metrics to mitigate the effects of calibration drift and changing risk across siloed datasets and time. In this study, we evaluated the effect of various combinations of Machine Learning (ML) models in improving model performance. The National Adult Cardiac Surgery Audit dataset was used (January 1996 to March 2019, 647,726 patients). We trained six different base learner models including Logistic Regression, Neuronetwork, Random Forest (RF), Weighted Support Vector Machine, Xgboost and Bayesian Update, based on two different variable sets of either Logistic EuroScore (LogES) or EuroScore II (ES II), partitioned by the time of score adoption (1996–2011 and 2012–2019). These base learner models are ensembled using nine different combinations to produce homogeneous or heterogeneous ensembles. Discrimination, calibration, clinical effectiveness and overall accuracy were assessed using an ensemble metric, referred to as clinical effectiveness metric (CEM). Xgboost homogenous ensemble (HE) was the highest performing model (CEM 0.725) with AUC (0.8327; 95% Confidence Interval (CI) 0.8323–0.8329) followed by Random Forest HE (CEM 0.723; AUC 0.8325; 95%CI 0.8320–0.8326). Across different heterogenous ensembles, significantly better performance was obtained by combining siloed datasets across time (CEM 0.720) than building ensembles of either 1996–2011 (t-test adjusted, p = 1.67e-6) or 2012–2019 (t-test adjusted, p = 1.35e-193) datasets alone. Both homogenous and heterogenous ML ensembles performed significantly better than traditional recalibration method (Bayesian Update). Combining the metrics covering all four aspects of discrimination, calibration, clinical usefulness and overall accuracy into a single ensemble metric improved the efficiency of cognitive decision-making. Xgboost/Random Forest homogenous ensembling and a highly heterogeneous ensemble approach showed high performance across multifaceted aspects of ML performance and were superior to traditional recalibration methods. Time-dependent ensemble combination of variables, having differing qualities according to time of score adoption, enabled previously siloed data to be combined, leading to increased power, clinical interpretability of variables and usage of data. For models to guide clinicians in individual decisions, performance exceeding these multifaceted benchmarks is necessary.
BibTeX
@inproceedings{angelini2022ensemble,
  author = {Gianni D. Angelini and Tim Dong and Shubhra Sinha and Bing Zhai and Daniel Fudulu and Arnaldo Dimagli and Pradeep Narayan and Jeremy Chan and Chris Holmes and Massimo Caputo and Lauren Dixon and Marco Gemelli and Hunaid A. Vohra and Umberto Benedetto},
  title = {{An Ensemble approach for Ensemble-Modelled Cardiac Surgery Risk Evaluation, Data Usage and Clinical Interpretability}},
  booktitle = {Research Square},
  year = {2022},
  doi = {10.21203/rs.3.rs-1905463/v1},
}
Figure