TL;DR
This study compares a new risk prediction model (IESG) for 90-day mortality after esophagectomy with machine learning (ML) approaches. While the IESG model offers adequate group-based risk stratification, ML models significantly outperformed it in accurately predicting individual patient risk, highlighting ML's potential for enhanced preoperative decision-making in complex surgeries.
What Problem Does This Paper Address?
Esophagectomy, a critical surgery for advanced esophageal cancer, carries substantial risks of postoperative complications and mortality. Accurate preoperative risk stratification is essential for informed patient-surgeon decision-making and optimizing surgical outcomes. While a new risk model, the International Esodata Study Group (IESG) model, has been proposed, its external validation and comparison with more advanced predictive techniques like machine learning (ML) are lacking. The challenge lies in capturing the complex, non-linear interactions between numerous patient and tumor factors that contribute to surgical risk, which traditional statistical models may struggle to fully encompass.
Methods
The researchers conducted a retrospective case-control study using data from 552 patients who underwent oncologic esophagectomy at two tertiary centers. They evaluated the existing IESG risk model by stratifying patients into five risk groups and comparing predicted vs. observed mortality. For the ML approach, they trained and validated various classifiers, including Decision Trees (DT), Logistic Regression (LR), Support Vector Machines (SVM), Gradient Boosting Machines (GBM), Random Forests (RF), and Neural Networks (NN).
Key methodological aspects include:
- Data Preprocessing: Handling missing values, normalizing numerical features, and one-hot encoding categorical features were standard ML practices.
- Model Development: A two-stage approach was used: an internal training/testing split (80:20) and then using the entire training cohort for final model development before testing on an independent external validation cohort.
- Feature Selection: Three methods were explored: all features, top 25 features based on ANOVA f-values, and features exceeding a median threshold in a support vector machine.
- Performance Metrics: The study employed Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), and Matthews Correlation Coefficient (MCC) to assess model discrimination. These metrics are crucial for evaluating model performance, especially when dealing with imbalanced datasets (where the outcome of interest, like mortality, is rare).
- Explainable AI (XAI): Shapley Additive Explanations (ShAP) analysis was used to understand the contribution of individual features to the ML models' predictions, enhancing interpretability.
Datasets
The study utilized a dataset of 552 patients who underwent oncologic esophagectomy between January 2009 and December 2021.
- Training Cohort: 409 patients from one tertiary center.
- Validation Cohort: 143 patients from a second, independent tertiary center.
This setup allowed for both internal validation (testing on unseen data from the same center) and external validation (testing on data from a different center), which is crucial for assessing the generalizability of the models. The dataset comprised 48 preoperative parameters, including demographic information, comorbidities (using the Charlson Comorbidity Index - CCI), tumor characteristics (TNM staging), histology, and neoadjuvant treatment details.
Key Insights and Ideas
- ML's Superiority in Individual Risk Prediction: The most striking insight is that ML models, despite being trained on a smaller subset of data compared to the IESG model's development, demonstrated significantly better performance in predicting individual patient risk of 90-day mortality. This suggests ML can uncover more nuanced patterns than traditional regression models.
- IESG Model's Utility for Group Stratification: While not as precise for individual predictions, the IESG model still provides an "adequate group-based risk prediction," making it a useful tool for general patient counseling and risk categorization.
- Importance of Non-Linear Relationships: The study implicitly highlights that the complex interplay of patient and tumor factors in surgical outcomes is not always linear, a domain where ML excels.
- Explainability Matters: The use of ShAP analysis to identify key predictive features (like INR and CCI) is vital. This "Explainable AI" approach builds trust and helps clinicians understand why a model makes a certain prediction, facilitating adoption.
- Preoperative Lab Values are Key: International Normalized Ratio (INR) and Charlson Comorbidity Index (CCI) consistently emerged as the most impactful features across multiple ML classifiers, reinforcing their known clinical significance and suggesting their critical role in predictive modeling.
Key Results
- IESG Model Performance: On the validation cohort, the IESG model achieved an AUROC of 0.44, an AUPRC of 0.11, and an MCC of 0.15. While it provided adequate group-based risk stratification, its discrimination performance was lower than expected.
- ML Model Performance (External Validation):
- AUROC: ML classifiers achieved significantly higher AUROCs, with the Random Forest (RF) classifier reaching 0.64 (95% CI: 0.63–0.65). This is a substantial improvement over the IESG model's 0.44.
- AUPRC: The Neural Network (NN) classifier achieved the highest AUPRC at 0.25 (95% CI: 0.24–0.27), compared to the IESG's 0.11.
- MCC: The RF classifier also showed the highest MCC at 0.27 (95% CI: 0.25–0.28), outperforming the IESG's 0.15.
- Feature Importance: INR was the most influential feature in six out of seven classifiers, followed by CCI in five classifiers. Other important factors included red blood cell count (RBC), ASA status, weight, and age.
Limitations and Future Work
- Sample Size: The authors acknowledge that the primary constraint is the sample size, which can challenge data-driven pattern recognition, especially with imbalanced classes. Larger datasets are needed for large-scale clinical implementation.
- Retrospective Design: The retrospective nature of the study may introduce data implications and potential biases.
- Heterogeneous Surgical Approaches: The study included patients who underwent open, laparoscopic, and robotic-assisted esophagectomies, which could introduce approach-related effects.
- Confounding Factors: Variations in chemotherapy doses and radiation doses were not fully accounted for, which could have influenced outcomes.
- Cohort Disparities: Minor disparities in patient characteristics between training and validation cohorts, though not statistically significant for 90-day mortality, could affect comparability.
Future Work: The authors suggest that future studies should focus on applying non-linear ML models to large-scale register data to fully evaluate their potential. They also emphasize the importance of using metrics that account for class imbalances and conducting external validation to accurately assess new AI models in this field.
Summary updated