TL;DR
This paper introduces a deep neural network (DNN) model capable of predicting multiple critical postoperative complications—mortality, acute kidney injury (AKI), and reintubation—using a single set of features available at the end of surgery. The DNN models demonstrated superior performance compared to the traditional ASA physical status score for most outcomes, highlighting the potential of machine learning in improving perioperative risk assessment.
What Problem Does This Paper Address?
Perioperative periods are inherently risky, with a significant percentage of surgical patients experiencing complications. Identifying high-risk patients is crucial for proactive management, but current risk scores often focus on single complications and may lack specificity. While machine learning (ML) has shown promise in integrating large datasets to predict multiple outcomes, there's a need to explore its effectiveness in a unified model for common perioperative events. This study aims to bridge this gap by developing and validating a DNN that can predict mortality, AKI, and reintubation simultaneously from a single feature set.
Methods
The researchers employed deep neural networks (DNNs) to build predictive models. They explored two main architectural approaches:
- Individual DNN models: Separate DNNs were trained to predict each outcome (mortality, AKI, reintubation) independently.
- Combined DNN model (Multitask Learning): A single DNN was trained to output probabilities for all three outcomes concurrently, hypothesizing that learning from one outcome could improve predictions for others.
They experimented with different feature sets:
- Original Feature Set (OFS): A set of 46 features extracted from electronic medical records (EMRs) at the end of surgery, plus six additional features related to intraoperative hypotension.
- Reduced Feature Set (RFS): Features with a Pearson correlation coefficient greater than 0.9 were removed from the OFS to mitigate multicollinearity.
- OFS + MAP features: The original feature set augmented with specific mean arterial pressure (MAP) features.
Model performance was evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC) as the primary metric, with F1 score, sensitivity, specificity, and precision as secondary outcomes. They also compared their DNN models against established benchmarks like the ASA physical status score, logistic regression (LR), Risk Stratification Index (RSI), and Risk Quantification Index (RQI).
Datasets
The study utilized data from 59,981 surgical procedures performed at UCLA between March 17, 2013, and July 16, 2016. This data was extracted from the Perioperative Data Warehouse (PDW), which aggregates information from the EMR system. A significant portion of patients (38,305) were excluded from AKI prediction due to missing creatinine values, leading to a smaller cohort for AKI-specific analyses. The data was split into training (80%) and testing (20%) sets.
Key Insights and Ideas
- Unified Prediction Power: A single set of perioperative features can effectively predict multiple distinct complications, suggesting a shared underlying risk profile or interconnectedness of these events.
- DNNs Outperform Traditional Scores: Deep neural networks, particularly for AKI and reintubation, demonstrated superior predictive accuracy compared to the widely used ASA physical status score. This underscores the potential of ML to enhance clinical decision-making beyond established heuristics.
- Multitask Learning Nuances: While the combined (multitask) DNN models showed comparable performance to individual DNNs, they didn't consistently outperform them. This suggests that while models can learn across outcomes, the benefit might be marginal or dependent on specific feature interactions.
- Feature Engineering Impact: Adding specific features related to intraoperative hypotension (MAP features) did not significantly improve model performance, nor did reducing the feature set. This implies that the initial feature set was already quite informative, or that there might be a limit to how much predictive power can be extracted from these specific data points.
- Metric Diversity is Key: The study highlights that no single metric (like AUC) tells the whole story. Different clinical applications might require optimizing for sensitivity, specificity, or precision, and the "best" model depends on its intended use.
Key Results
- AUC Performance: The best DNN models achieved AUCs of 0.792 for AKI, 0.879 for reintubation, 0.907 for mortality, and 0.874 for any outcome.
- Comparison to ASA: DNN models outperformed the ASA score for AKI, reintubation, and any outcome. For mortality, the best DNN model achieved a slightly higher AUC (0.907) than the ASA score (0.839), though the paper notes that no model outperformed the ASA score or RQI for mortality in all comparisons.
- Feature Set Impact: Adding MAP features or reducing the feature set did not consistently improve performance across all models and outcomes.
- McNemar Test: Statistical comparisons (McNemar test) indicated significant differences in error rates between LR and DNN models for AKI, mortality, and any outcome, with DNNs generally performing better.
Limitations and Future Work
- Single-Center Study: The models were developed and validated at a single institution, which may limit their generalizability to other healthcare settings. Future work should involve multi-center validation.
- Data Loss for AKI: A substantial number of patients were excluded from AKI analysis due to missing creatinine data, potentially biasing the AKI model results.
- Feature Set Size: The study used a relatively small feature set (around 50 features) compared to typical ML applications that might use thousands or millions. While this was intentional for clinical interpretability, exploring larger feature sets could be a future direction.
- Model Interpretability: While DNNs showed strong predictive power, the "black box" nature of these models can be a limitation for clinical adoption. Future work could focus on enhancing the interpretability of these models.
- Threshold Optimization: The paper emphasizes that the choice of threshold is critical and depends on the specific clinical application (e.g., screening vs. treatment decision). Further research could explore adaptive thresholding strategies.
Summary updated