Interpretable Machine Learning Model for Early Prediction of Mortality in Elderly Patients with Multiple Organ Dysfunction Syndrome (MODS): a Multicenter Retrospective Study and Cross Validation

Xiaoli Liu, Pan Hu, Zhi Mao, Po-Chih Kuo, Peiyao Li, Chao Liu, Jie Hu, Deyu Li, Desen Cao, Roger G. Mark, Leo Anthony Celi, Zhengbo Zhang, Feihu ZhouarXiv 2020 · OpenReview

TL;DR

This paper introduces an interpretable machine learning model, XGBoost with SHAP, for predicting mortality in elderly ICU patients with Multiple Organ Dysfunction Syndrome (MODS). The model demonstrates superior performance compared to existing clinical scores and baseline machine learning models across multiple datasets, offering clinicians actionable insights into patient risk.

What Problem Does This Paper Address?

Elderly patients in the Intensive Care Unit (ICU) with Multiple Organ Dysfunction Syndrome (MODS) face a high risk of mortality. Early detection and accurate prognosis are crucial for timely intervention and improved outcomes. However, current clinical scoring systems often lack sufficient sensitivity and specificity, failing to accurately guide clinical decisions. This study aims to bridge this gap by developing a more accurate, interpretable, and generalizable predictive model for this vulnerable patient population.

Methods

The researchers employed the eXtreme Gradient Boosting (XGBoost) algorithm, a powerful tree-based ensemble method known for its speed and predictive accuracy. To address the "black box" nature of many machine learning models, they integrated SHapley Additive exPlanations (SHAP). SHAP provides a unified framework to explain the predictions of machine learning models by assigning an importance value to each feature for each individual prediction. This allows clinicians to understand why the model makes a particular prediction, identifying key contributing factors to a patient's mortality risk. The study explored different data source combinations (single-center, multi-center, and fused) to ensure model generalizability.

Datasets

The study utilized three large, publicly available ICU datasets:

  • MIMIC-III: A single-center dataset from Beth Israel Deaconess Medical Center (57,786 unique ICU admissions).
  • eICU-CRD: A multi-center, telehealth ICU dataset from 208 Critical Care Units in the US (200,859 unique ICU admissions).
  • PLAGH-S: A surgical ICU database from a tertiary hospital in China (1,462 unique ICU admissions).

These datasets were used for model generation and rigorous internal and external validation, ensuring the model's robustness across different healthcare settings and patient populations.

Key Insights and Ideas

  • Interpretability is Key: The study strongly emphasizes that a model's predictive power is only part of the story. Providing clinicians with an understanding of why a prediction is made (via SHAP) is crucial for trust and effective clinical integration.
  • Beyond Standard Scores: The model identified factors like Blood Urea Nitrogen (BUN) and shock index (SI) as highly predictive of mortality, which are often overlooked or less emphasized in traditional clinical scores. This suggests a need to re-evaluate the importance of certain biomarkers.
  • Nervous System and Respiratory System Dominance: Glasgow Coma Scale (GCS) and respiratory rate were found to be among the most critical predictors, highlighting the significant impact of neurological and respiratory status on mortality risk in MODS patients.
  • Robustness Across Diverse Datasets: The model's consistent performance across US and Chinese datasets, with no overlap between training and testing sets, demonstrates its generalizability and potential for broad clinical application.

Key Results

The developed interpretable model (Model 3, using fused MIMIC-III and eICU-CRD data) achieved strong predictive performance:

  • MIMIC-III validation: AUC of 0.858, sensitivity of 0.834, specificity of 0.705.
  • eICU-CRD validation: AUC of 0.849, sensitivity of 0.763, specificity of 0.784.
  • PLAGH-S validation: AUC of 0.838, sensitivity of 0.882, specificity of 0.691.

Crucially, this model significantly outperformed established clinical scores (OASIS, APSIII, MODS, SAPS, SOFA) and baseline machine learning models (LR, NN, SVM, RF, NB) in predicting hospital mortality. The top identified risk factors included GCS, respiratory rate, BUN, age, and shock index.

Limitations and Future Work

The authors acknowledge several limitations:

  • Real-time Prediction: The current model focuses on the initial 24 hours of ICU stay. Future work could involve developing real-time prediction models using time-series data, potentially with Long Short-Term Memory (LSTM) networks.
  • PLAGH-S Dataset Specificity: The PLAGH-S dataset is from a single surgical ICU, which may not fully represent the diversity of ICU types found in MIMIC-III and eICU-CRD. Future efforts will focus on standardizing data and analyzing a larger, more diverse cohort from the PLAGH General Intensive Care Unit Database.
  • Data Completeness: While the model uses a comprehensive set of features, some data points might be missing or inconsistently recorded across different databases.

The study's success in developing a robust, interpretable, and generalizable model for a critical patient population lays a strong foundation for future advancements in AI-driven clinical decision support.

Summary updated

Abstract

Background: Elderly patients with MODS have high risk of death and poor prognosis. The performance of current scoring systems assessing the severity of MODS and its mortality remains unsatisfactory. This study aims to develop an interpretable and generalizable model for early mortality prediction in elderly patients with MODS. Methods: The MIMIC-III, eICU-CRD and PLAGH-S databases were employed for model generation and evaluation. We used the eXtreme Gradient Boosting model with the SHapley Additive exPlanations method to conduct early and interpretable predictions of patients' hospital outcome. Three types of data source combinations and five typical evaluation indexes were adopted to develop a generalizable model. Findings: The interpretable model, with optimal performance developed by using MIMIC-III and eICU-CRD datasets, was separately validated in MIMIC-III, eICU-CRD and PLAGH-S datasets (no overlapping with training set). The performances of the model in predicting hospital mortality as validated by the three datasets were: AUC of 0.858, sensitivity of 0.834 and specificity of 0.705; AUC of 0.849, sensitivity of 0.763 and specificity of 0.784; and AUC of 0.838, sensitivity of 0.882 and specificity of 0.691, respectively. Comparisons of AUC between this model and baseline models with MIMIC-III dataset validation showed superior performances of this model; In addition, comparisons in AUC between this model and commonly used clinical scores showed significantly better performance of this model. Interpretation: The interpretable machine learning model developed in this study using fused datasets with large sample sizes was robust and generalizable. This model outperformed the baseline models and several clinical scores for early prediction of mortality in elderly ICU patients. The interpretative nature of this model provided clinicians with the ranking of mortality risk features.
BibTeX
@misc{liu2020interpretable,
  author = {Xiaoli Liu and Pan Hu and Zhi Mao and Po-Chih Kuo and Peiyao Li and Chao Liu and Jie Hu and Deyu Li and Desen Cao and Roger G. Mark and Leo Anthony Celi and Zhengbo Zhang and Feihu Zhou},
  title = {{Interpretable Machine Learning Model for Early Prediction of Mortality in Elderly Patients with Multiple Organ Dysfunction Syndrome (MODS): a Multicenter Retrospective Study and Cross Validation}},
  year = {2020},
  eprint = {2001.10977},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.10977},
}
Figure