Machine Learning Prediction Models for Mechanically Ventilated Patients: Analyses of the MIMIC-III Database

Yibing Zhu, Jin Zhang, Guowei Wang, Ren-qi Yao, Chao Ren, Chen Ge, Xin Jin, Junyang Guo, Shi Liu, Hua Zheng, Yan Chen, Qianqian Guo, Lin Li, Bin Du, Xiuming Xi, Wei Li, Hui‐Bin Huang, Yang Li, Qian YuFrontiers in Medicine 2021 · DOI

TL;DR

This study leverages machine learning on early ICU data to predict mortality in mechanically ventilated patients, outperforming existing scores. The XGBoost model demonstrated the best predictive performance, identifying age, respiratory dysfunction, SAPS II, maximum hemoglobin, and minimum lactate as key risk factors.

What Problem Does This Paper Address?

Mechanically ventilated patients in the Intensive Care Unit (ICU) face a high risk of mortality. While established scoring systems like SAPS II, OASIS, and SOFA exist for general ICU populations, there's a need for more precise prediction models specifically for mechanically ventilated patients, utilizing readily available early-stage data. The challenge lies in identifying which early clinical features are most predictive of mortality in this vulnerable group and developing robust models to capture these relationships.

Methods

The researchers employed a retrospective study design using the MIMIC-III database. They trained and evaluated seven different machine learning models: k-nearest neighbors (KNN), logistic regression, bagging, decision tree, random forest, Extreme Gradient Boosting (XGBoost), and neural network. A 70/30 split was used for training and testing the models. Model performance was assessed using Area Under the Receiver Operating Characteristic Curves (AUCs) and calibration plots. The significance of predictors was identified using model analysis and Shapley Additive Explanation (SHAP) values.

Datasets

The study utilized the Medical Information Mart for Intensive Care (MIMIC-III) database. This is a publicly available, large, single-center database containing de-identified health-related data for over 38,000 patients admitted to intensive care units at Beth Israel Deaconess Medical Center between 2001 and 2012. No new benchmarks were introduced; the study focused on analyzing existing data with advanced ML techniques.

Key Insights and Ideas

  • Early Predictors Matter: The study strongly suggests that crucial information for predicting mortality in mechanically ventilated patients is available within the first 24 hours of ICU admission. This highlights the potential for early intervention and risk stratification.
  • XGBoost Dominance: Among the tested models, XGBoost consistently outperformed others, underscoring its effectiveness in handling complex, high-dimensional clinical data for predictive tasks.
  • Beyond Traditional Scores: While established scores like SAPS II are important, the study demonstrates that combining them with a broader set of early clinical features (demographics, vital signs, lab results) can significantly improve predictive accuracy.
  • Specific Risk Factors Identified: The identification of age, respiratory dysfunction, SAPS II, maximum hemoglobin, and minimum lactate as top predictors provides actionable insights for clinicians. For instance, understanding the role of hemoglobin and lactate levels early on could guide treatment decisions.

Key Results

  • A total of 25,659 adult patients were included in the final analysis after data preprocessing.
  • The mortality rate for the cohort was 45.5%.
  • The XGBoost model achieved the highest AUC of 0.821 on the testing set, outperforming other models like logistic regression (0.818) and random forest (0.819).
  • Calibration plots indicated that most models, except the neural network, performed well in aligning predicted risks with observed outcomes.
  • The top five predictors identified by the XGBoost model were: age (0.410), respiratory dysfunction (0.309), SAPS II score (0.302), maximum hemoglobin (0.209), and minimum lactate (0.194).

Limitations and Future Work

The authors acknowledge several limitations:

  • Single-Center Data: The study relied on data from a single institution (MIMIC-III), which may limit the generalizability of the findings to other healthcare settings.
  • Retrospective Design: The retrospective nature of the study means that external validation is crucial to confirm the models' performance in different populations.
  • Missing Data: While efforts were made to handle missing data, some variables with extensive missingness were excluded, potentially impacting the completeness of the analysis.
  • Focus on Hospital Mortality: The study primarily focused on hospital mortality. Future work could expand to include other important outcomes like ventilator-free days or long-term mortality.
  • Withdrawal of Care: Patients who were withdrawn from care were not explicitly excluded, which could introduce bias.

Future work should focus on external validation of the developed models and predictors in diverse patient populations and healthcare systems. Investigating other outcome measures and refining the handling of missing data are also suggested avenues.

Summary updated

Abstract

<b>Background:</b> Mechanically ventilated patients in the intensive care unit (ICU) have high mortality rates. There are multiple prediction scores, such as the Simplified Acute Physiology Score II (SAPS II), Oxford Acute Severity of Illness Score (OASIS), and Sequential Organ Failure Assessment (SOFA), widely used in the general ICU population. We aimed to establish prediction scores on mechanically ventilated patients with the combination of these disease severity scores and other features available on the first day of admission. <b>Methods:</b> A retrospective administrative database study from the Medical Information Mart for Intensive Care (MIMIC-III) database was conducted. The exposures of interest consisted of the demographics, pre-ICU comorbidity, ICU diagnosis, disease severity scores, vital signs, and laboratory test results on the first day of ICU admission. Hospital mortality was used as the outcome. We used the machine learning methods of <i>k</i>-nearest neighbors (KNN), logistic regression, bagging, decision tree, random forest, Extreme Gradient Boosting (XGBoost), and neural network for model establishment. A sample of 70% of the cohort was used for the training set; the remaining 30% was applied for testing. Areas under the receiver operating characteristic curves (AUCs) and calibration plots would be constructed for the evaluation and comparison of the models' performance. The significance of the risk factors was identified through models and the top factors were reported. <b>Results:</b> A total of 28,530 subjects were enrolled through the screening of the MIMIC-III database. After data preprocessing, 25,659 adult patients with 66 predictors were included in the model analyses. With the training set, the models of KNN, logistic regression, decision tree, random forest, neural network, bagging, and XGBoost were established and the testing set obtained AUCs of 0.806, 0.818, 0.743, 0.819, 0.780, 0.803, and 0.821, respectively. The calibration curves of all the models, except for the neural network, performed well. The XGBoost model performed best among the seven models. The top five predictors were age, respiratory dysfunction, SAPS II score, maximum hemoglobin, and minimum lactate. <b>Conclusion:</b> The current study indicates that models with the risk of factors on the first day could be successfully established for predicting mortality in ventilated patients. The XGBoost model performs best among the seven machine learning models.
BibTeX
@inproceedings{zhu2021machine,
  author = {Yibing Zhu and Jin Zhang and Guowei Wang and Ren-qi Yao and Chao Ren and Chen Ge and Xin Jin and Junyang Guo and Shi Liu and Hua Zheng and Yan Chen and Qianqian Guo and Lin Li and Bin Du and Xiuming Xi and Wei Li and Hui‐Bin Huang and Yang Li and Qian Yu},
  title = {{Machine Learning Prediction Models for Mechanically Ventilated Patients: Analyses of the MIMIC-III Database}},
  booktitle = {Frontiers in Medicine},
  year = {2021},
  doi = {10.3389/fmed.2021.662340},
}
Figure