Development and validation of a deep neural network model to predict postoperative mortality, acute kidney injury, and reintubation using a single feature set

Ira Hofer, Christine K. Lee, Eilon Gabel, Pierre Baldi, Maxime CannessonNpj Digit. Medicine 2020 · DOI · OpenReview

TL;DR

This paper introduces a deep neural network (DNN) model capable of predicting multiple critical postoperative complications—mortality, acute kidney injury (AKI), and reintubation—using a single set of features available at the end of surgery. The DNN models demonstrated superior performance compared to the traditional ASA physical status score for most outcomes, highlighting the potential of machine learning in improving perioperative risk assessment.

What Problem Does This Paper Address?

Perioperative periods are inherently risky, with a significant percentage of surgical patients experiencing complications. Identifying high-risk patients is crucial for proactive management, but current risk scores often focus on single complications and may lack specificity. While machine learning (ML) has shown promise in integrating large datasets to predict multiple outcomes, there's a need to explore its effectiveness in a unified model for common perioperative events. This study aims to bridge this gap by developing and validating a DNN that can predict mortality, AKI, and reintubation simultaneously from a single feature set.

Methods

The researchers employed deep neural networks (DNNs) to build predictive models. They explored two main architectural approaches:

  • Individual DNN models: Separate DNNs were trained to predict each outcome (mortality, AKI, reintubation) independently.
  • Combined DNN model (Multitask Learning): A single DNN was trained to output probabilities for all three outcomes concurrently, hypothesizing that learning from one outcome could improve predictions for others.

They experimented with different feature sets:

  • Original Feature Set (OFS): A set of 46 features extracted from electronic medical records (EMRs) at the end of surgery, plus six additional features related to intraoperative hypotension.
  • Reduced Feature Set (RFS): Features with a Pearson correlation coefficient greater than 0.9 were removed from the OFS to mitigate multicollinearity.
  • OFS + MAP features: The original feature set augmented with specific mean arterial pressure (MAP) features.

Model performance was evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC) as the primary metric, with F1 score, sensitivity, specificity, and precision as secondary outcomes. They also compared their DNN models against established benchmarks like the ASA physical status score, logistic regression (LR), Risk Stratification Index (RSI), and Risk Quantification Index (RQI).

Datasets

The study utilized data from 59,981 surgical procedures performed at UCLA between March 17, 2013, and July 16, 2016. This data was extracted from the Perioperative Data Warehouse (PDW), which aggregates information from the EMR system. A significant portion of patients (38,305) were excluded from AKI prediction due to missing creatinine values, leading to a smaller cohort for AKI-specific analyses. The data was split into training (80%) and testing (20%) sets.

Key Insights and Ideas

  • Unified Prediction Power: A single set of perioperative features can effectively predict multiple distinct complications, suggesting a shared underlying risk profile or interconnectedness of these events.
  • DNNs Outperform Traditional Scores: Deep neural networks, particularly for AKI and reintubation, demonstrated superior predictive accuracy compared to the widely used ASA physical status score. This underscores the potential of ML to enhance clinical decision-making beyond established heuristics.
  • Multitask Learning Nuances: While the combined (multitask) DNN models showed comparable performance to individual DNNs, they didn't consistently outperform them. This suggests that while models can learn across outcomes, the benefit might be marginal or dependent on specific feature interactions.
  • Feature Engineering Impact: Adding specific features related to intraoperative hypotension (MAP features) did not significantly improve model performance, nor did reducing the feature set. This implies that the initial feature set was already quite informative, or that there might be a limit to how much predictive power can be extracted from these specific data points.
  • Metric Diversity is Key: The study highlights that no single metric (like AUC) tells the whole story. Different clinical applications might require optimizing for sensitivity, specificity, or precision, and the "best" model depends on its intended use.

Key Results

  • AUC Performance: The best DNN models achieved AUCs of 0.792 for AKI, 0.879 for reintubation, 0.907 for mortality, and 0.874 for any outcome.
  • Comparison to ASA: DNN models outperformed the ASA score for AKI, reintubation, and any outcome. For mortality, the best DNN model achieved a slightly higher AUC (0.907) than the ASA score (0.839), though the paper notes that no model outperformed the ASA score or RQI for mortality in all comparisons.
  • Feature Set Impact: Adding MAP features or reducing the feature set did not consistently improve performance across all models and outcomes.
  • McNemar Test: Statistical comparisons (McNemar test) indicated significant differences in error rates between LR and DNN models for AKI, mortality, and any outcome, with DNNs generally performing better.

Limitations and Future Work

  • Single-Center Study: The models were developed and validated at a single institution, which may limit their generalizability to other healthcare settings. Future work should involve multi-center validation.
  • Data Loss for AKI: A substantial number of patients were excluded from AKI analysis due to missing creatinine data, potentially biasing the AKI model results.
  • Feature Set Size: The study used a relatively small feature set (around 50 features) compared to typical ML applications that might use thousands or millions. While this was intentional for clinical interpretability, exploring larger feature sets could be a future direction.
  • Model Interpretability: While DNNs showed strong predictive power, the "black box" nature of these models can be a limitation for clinical adoption. Future work could focus on enhancing the interpretability of these models.
  • Threshold Optimization: The paper emphasizes that the choice of threshold is critical and depends on the specific clinical application (e.g., screening vs. treatment decision). Further research could explore adaptive thresholding strategies.

Summary updated

Abstract

During the perioperative period patients often suffer complications, including acute kidney injury (AKI), reintubation, and mortality. In order to effectively prevent these complications, high-risk patients must be readily identified. However, most current risk scores are designed to predict a single postoperative complication and often lack specificity on the patient level. In other fields, machine learning (ML) has been shown to successfully create models to predict multiple end points using a single input feature set. We hypothesized that ML can be used to create models to predict postoperative mortality, AKI, reintubation, and a combined outcome using a single set of features available at the end of surgery. A set of 46 features available at the end of surgery, including drug dosing, blood loss, vital signs, and others were extracted. Additionally, six additional features accounting for total intraoperative hypotension were extracted and trialed for different models. A total of 59,981 surgical procedures met inclusion criteria and the deep neural networks (DNN) were trained on 80% of the data, with 20% reserved for testing. The network performances were then compared to ASA Physical Status. In addition to creating separate models for each outcome, a multitask learning model was trialed that used information on all outcomes to predict the likelihood of each outcome individually. The overall rate of the examined complications in this data set was 0.79% for mortality, 22.3% (of 21,676 patients with creatinine values) for AKI, and 1.1% for reintubation. Overall, there was significant overlap between the various model types for each outcome, with no one modeling technique consistently performing the best. However, the best DNN models did beat the ASA score for all outcomes other than mortality. The highest area under the receiver operating characteristic curve (AUC) models were 0.792 (0.775–0.808) for AKI, 0.879 (0.851–0.905) for reintubation, 0.907 (0.872–0.938) for mortality, and 0.874 (0.864–0.866) for any outcome. The ASA score alone achieved AUCs of 0.652 (0.636–0.669) for AKI, 0.787 (0.757–0.818) for reintubation, 0.839 (0.804–0.875) for mortality, and 0.76 (0.748–0.773) for any outcome. Overall, the DNN architecture was able to create models that outperformed the ASA physical status to predict all outcomes based on a single feature set, consisting of objective data available at the end of surgery. No one model architecture consistently performed the best.
BibTeX
@article{hofer2020development,
  author = {Ira Hofer and Christine K. Lee and Eilon Gabel and Pierre Baldi and Maxime Cannesson},
  title = {{Development and validation of a deep neural network model to predict postoperative mortality, acute kidney injury, and reintubation using a single feature set}},
  journal = {Npj Digit. Medicine},
  year = {2020},
  doi = {10.1038/s41746-020-0248-0},
}
Figure