A Simple Weaning Model Based on Interpretable Machine Learning Algorithm for Patients With Sepsis: A Research of MIMIC-IV and eICU Databases

Wanjun Liu, Tao Gan, Yijun Zhang, Wenyan Xiao, Jin Zhang, Yu Liu, Zongqing Lu, Tianfeng Hua, Min YangFrontiers in Medicine 2022 · DOI

TL;DR

This paper introduces a novel, interpretable machine learning model to predict successful weaning from mechanical ventilation in sepsis patients. By leveraging large clinical datasets and the XGBoost algorithm, the model achieves high predictive accuracy and has been developed into a user-friendly web tool for clinical application.

What Problem Does This Paper Address?

Mechanical ventilation is a critical intervention for patients with sepsis, but prolonged reliance on it can lead to poor prognoses. Conversely, premature weaning can also be detrimental. Currently, there's a lack of specific, reliable tools to accurately predict successful weaning in sepsis patients, who may have unique physiological responses. This gap makes it challenging for clinicians to make timely and appropriate decisions about ventilator weaning, potentially impacting patient outcomes, including mortality and ICU length of stay.

Methods

The researchers employed a data-driven approach using machine learning to develop their weaning prediction model. They extracted a comprehensive set of clinical variables from patient records within 24 hours prior to weaning. To build and validate their model, they utilized the eXtremely Gradient Boosting (XGBoost) algorithm, a powerful ensemble method known for its predictive accuracy.

Key methodological steps included:

  • Feature Selection: Variables were initially selected based on clinical importance and further refined using Shapley Additive Explanations (SHAP) values to identify the most impactful predictors.
  • Model Training and Validation: The MIMIC-IV dataset was split into training and internal validation sets, while the eICU-CRD dataset served as an independent external validation set. This multi-dataset approach ensures the model's generalizability.
  • Model Simplification: After identifying the best-performing model (XGBoost), the researchers aimed to create a simpler, more clinically applicable version. This was achieved through techniques like recursive feature elimination (RFE), ultimately reducing the model to just four key variables.
  • Interpretability: SHAP values were used to explain the model's predictions, providing insights into which variables contribute most to the probability of successful weaning and how their values influence the outcome.
  • Web-Based Tool Development: The simplified model was translated into a practical web-based tool, making it accessible for clinicians to use in real-time decision-making.

Datasets

The study utilized two large, publicly available critical care datasets:

  • MIMIC-IV (Medical Information Mart for Intensive Care IV): This database contains over 40,000 ICU patients from Beth Israel Deaconess Medical Center (2008-2019). It was used for both training and internal validation.
  • eICU-CRD (eICU Collaborative Research Database): This is a multicenter database with over 200,000 ICU admissions in the United States. It served as an independent external validation set, crucial for assessing the model's generalizability across different patient populations and healthcare settings.

Key Insights and Ideas

  • Weaning Success is a Strong Predictor of Outcomes: The study strongly reinforces that successful weaning is independently associated with significantly reduced 28-day mortality and shorter ICU stays in sepsis patients. This highlights the critical importance of accurate weaning prediction.
  • XGBoost Outperforms Traditional Models: The XGBoost algorithm demonstrated superior predictive performance (higher AUROC) compared to other common machine learning models (like Logistic Regression, SVM, etc.) and traditional weaning indices (CROP, RSBI) in both internal and external validation.
  • Interpretability Enhances Clinical Trust: The use of SHAP values to explain the model's predictions is a significant contribution. It moves beyond a "black box" approach, allowing clinicians to understand why the model makes a certain prediction, fostering trust and facilitating adoption.
  • Simplified Model Retains High Predictive Power: The development of a simplified model with only four variables, while showing a slight decrease in predictive performance, still maintains excellent accuracy. This trade-off between complexity and performance is key for practical clinical tools.
  • Key Predictors Emerge: The study identified specific variables that are highly influential in predicting weaning success, such as IMV duration, highest PEEP level, urine output, and lowest base excess. Understanding these factors can guide clinical management.

Key Results

  • Model Performance: The full XGBoost model achieved an AUROC of 0.80 in the internal validation set and an impressive 0.86 in the external validation set.
  • Simplified Model Performance: The simplified four-variable model also demonstrated strong predictive capabilities, with an AUROC of 0.75 in the internal validation set and 0.78 in the external validation set.
  • Variable Importance: The study identified IMV duration, highest PEEP level, urine output, and lowest base excess as the most important predictors in the simplified model.
  • Clinical Outcomes: Weaning success was significantly associated with lower 28-day mortality (p < 0.001) and shorter ICU length of stay (p = 0.002).
  • Web Tool: A web-based tool has been developed to provide clinicians with easy access to the simplified weaning prediction model.

Limitations and Future Work

  • Exclusion of Key Indicators: The study acknowledges that comprehensive indicators like SOFA and SAPS, which could potentially improve predictive performance, were not included in the final model due to data availability or the focus on simpler, more readily available variables.
  • Generalizability: While validated on two large datasets, the authors suggest the need for additional data sources to further demonstrate the model's generalizability across diverse patient populations and healthcare systems.
  • False Positives/Negatives: Despite good performance, the model still has a degree of false-positive and false-negative rates (PPV and NPV were around 0.75-0.81 and 0.62-0.77, respectively). The authors suggest that incorporating more valuable variables and dynamic prediction models could improve performance.
  • Potential for Lactate and Central Venous Pressure: The paper notes that variables like lactate and central venous pressure, which were excluded due to high error rates, might hold predictive value and could be explored in future research.
  • Dynamic Prediction: The current model uses data from a specific time point. Future work could focus on developing dynamic prediction models that incorporate changes over time.

Summary updated

Abstract

Weaning success is independently related to short-term mortality in patients with sepsis. The simplified model based on the XGBoost algorithm provides good predictive performance and great clinical applicablity for weaning, and a web-based tool was developed for better clinical application.
BibTeX
@inproceedings{liu2022simple,
  author = {Wanjun Liu and Tao Gan and Yijun Zhang and Wenyan Xiao and Jin Zhang and Yu Liu and Zongqing Lu and Tianfeng Hua and Min Yang},
  title = {{A Simple Weaning Model Based on Interpretable Machine Learning Algorithm for Patients With Sepsis: A Research of MIMIC-IV and eICU Databases}},
  booktitle = {Frontiers in Medicine},
  year = {2022},
  doi = {10.3389/fmed.2021.814566},
}
Figure