Enhancing Preoperative Outcome Prediction: A Comparative Retrospective Case–Control Study on Machine Learning versus the International Esodata Study Group Risk Model for Predicting 90-Day Mortality in Oncologic Esophagectomy

Axel Winter, Robin van de Water, Bjarne Pfitzner, Marius Ibach, Christoph Riepe, Robert Ahlborn, Lara Faraj, Felix Krenzien, Eva Maria Dobrindt, Jonas Raakow, Igor M. Sauer, Bert Arnrich, Katharina Beyer, Christian Denecke, Johann Pratschke, Max M. MaurerCancers 2024 · DOI

TL;DR

This study compares a new risk prediction model (IESG) for 90-day mortality after esophagectomy with machine learning (ML) approaches. While the IESG model offers adequate group-based risk stratification, ML models significantly outperformed it in accurately predicting individual patient risk, highlighting ML's potential for enhanced preoperative decision-making in complex surgeries.

What Problem Does This Paper Address?

Esophagectomy, a critical surgery for advanced esophageal cancer, carries substantial risks of postoperative complications and mortality. Accurate preoperative risk stratification is essential for informed patient-surgeon decision-making and optimizing surgical outcomes. While a new risk model, the International Esodata Study Group (IESG) model, has been proposed, its external validation and comparison with more advanced predictive techniques like machine learning (ML) are lacking. The challenge lies in capturing the complex, non-linear interactions between numerous patient and tumor factors that contribute to surgical risk, which traditional statistical models may struggle to fully encompass.

Methods

The researchers conducted a retrospective case-control study using data from 552 patients who underwent oncologic esophagectomy at two tertiary centers. They evaluated the existing IESG risk model by stratifying patients into five risk groups and comparing predicted vs. observed mortality. For the ML approach, they trained and validated various classifiers, including Decision Trees (DT), Logistic Regression (LR), Support Vector Machines (SVM), Gradient Boosting Machines (GBM), Random Forests (RF), and Neural Networks (NN).

Key methodological aspects include:

  • Data Preprocessing: Handling missing values, normalizing numerical features, and one-hot encoding categorical features were standard ML practices.
  • Model Development: A two-stage approach was used: an internal training/testing split (80:20) and then using the entire training cohort for final model development before testing on an independent external validation cohort.
  • Feature Selection: Three methods were explored: all features, top 25 features based on ANOVA f-values, and features exceeding a median threshold in a support vector machine.
  • Performance Metrics: The study employed Area Under the Receiver Operating Characteristic Curve (AUROC), Area Under the Precision-Recall Curve (AUPRC), and Matthews Correlation Coefficient (MCC) to assess model discrimination. These metrics are crucial for evaluating model performance, especially when dealing with imbalanced datasets (where the outcome of interest, like mortality, is rare).
  • Explainable AI (XAI): Shapley Additive Explanations (ShAP) analysis was used to understand the contribution of individual features to the ML models' predictions, enhancing interpretability.

Datasets

The study utilized a dataset of 552 patients who underwent oncologic esophagectomy between January 2009 and December 2021.

  • Training Cohort: 409 patients from one tertiary center.
  • Validation Cohort: 143 patients from a second, independent tertiary center.

This setup allowed for both internal validation (testing on unseen data from the same center) and external validation (testing on data from a different center), which is crucial for assessing the generalizability of the models. The dataset comprised 48 preoperative parameters, including demographic information, comorbidities (using the Charlson Comorbidity Index - CCI), tumor characteristics (TNM staging), histology, and neoadjuvant treatment details.

Key Insights and Ideas

  • ML's Superiority in Individual Risk Prediction: The most striking insight is that ML models, despite being trained on a smaller subset of data compared to the IESG model's development, demonstrated significantly better performance in predicting individual patient risk of 90-day mortality. This suggests ML can uncover more nuanced patterns than traditional regression models.
  • IESG Model's Utility for Group Stratification: While not as precise for individual predictions, the IESG model still provides an "adequate group-based risk prediction," making it a useful tool for general patient counseling and risk categorization.
  • Importance of Non-Linear Relationships: The study implicitly highlights that the complex interplay of patient and tumor factors in surgical outcomes is not always linear, a domain where ML excels.
  • Explainability Matters: The use of ShAP analysis to identify key predictive features (like INR and CCI) is vital. This "Explainable AI" approach builds trust and helps clinicians understand why a model makes a certain prediction, facilitating adoption.
  • Preoperative Lab Values are Key: International Normalized Ratio (INR) and Charlson Comorbidity Index (CCI) consistently emerged as the most impactful features across multiple ML classifiers, reinforcing their known clinical significance and suggesting their critical role in predictive modeling.

Key Results

  • IESG Model Performance: On the validation cohort, the IESG model achieved an AUROC of 0.44, an AUPRC of 0.11, and an MCC of 0.15. While it provided adequate group-based risk stratification, its discrimination performance was lower than expected.
  • ML Model Performance (External Validation):
    • AUROC: ML classifiers achieved significantly higher AUROCs, with the Random Forest (RF) classifier reaching 0.64 (95% CI: 0.63–0.65). This is a substantial improvement over the IESG model's 0.44.
  • AUPRC: The Neural Network (NN) classifier achieved the highest AUPRC at 0.25 (95% CI: 0.24–0.27), compared to the IESG's 0.11.
  • MCC: The RF classifier also showed the highest MCC at 0.27 (95% CI: 0.25–0.28), outperforming the IESG's 0.15.
  • Feature Importance: INR was the most influential feature in six out of seven classifiers, followed by CCI in five classifiers. Other important factors included red blood cell count (RBC), ASA status, weight, and age.

Limitations and Future Work

  • Sample Size: The authors acknowledge that the primary constraint is the sample size, which can challenge data-driven pattern recognition, especially with imbalanced classes. Larger datasets are needed for large-scale clinical implementation.
  • Retrospective Design: The retrospective nature of the study may introduce data implications and potential biases.
  • Heterogeneous Surgical Approaches: The study included patients who underwent open, laparoscopic, and robotic-assisted esophagectomies, which could introduce approach-related effects.
  • Confounding Factors: Variations in chemotherapy doses and radiation doses were not fully accounted for, which could have influenced outcomes.
  • Cohort Disparities: Minor disparities in patient characteristics between training and validation cohorts, though not statistically significant for 90-day mortality, could affect comparability.

Future Work: The authors suggest that future studies should focus on applying non-linear ML models to large-scale register data to fully evaluate their potential. They also emphasize the importance of using metrics that account for class imbalances and conducting external validation to accurately assess new AI models in this field.

Summary updated

Abstract

Risk prediction prior to oncologic esophagectomy is crucial for assisting surgeons and patients in their joint informed decision making. Recently, a new risk prediction model for 90-day mortality after esophagectomy using the International Esodata Study Group (IESG) database was proposed, allowing for the preoperative assignment of patients into different risk categories. However, given the non-linear dependencies between patient- and tumor-related risk factors contributing to cumulative surgical risk, machine learning (ML) may evolve as a novel and more integrated approach for mortality prediction. We evaluated the IESG risk model and compared its performance to ML models. Multiple classifiers were trained and validated on 552 patients from two independent centers undergoing oncologic esophagectomies. The discrimination performance of each model was assessed utilizing the area under the receiver operating characteristics curve (AUROC), the area under the precision-recall curve (AUPRC), and the Matthews correlation coefficient (MCC). The 90-day mortality rate was 5.8%. We found that IESG categorization allowed for adequate group-based risk prediction. However, ML models provided better discrimination performance, reaching superior AUROCs (0.64 [0.63-0.65] vs. 0.44 [0.32-0.56]), AUPRCs (0.25 [0.24-0.27] vs. 0.11 [0.05-0.21]), and MCCs (0.27 ([0.25-0.28] vs. 0.15 [0.03-0.27]). Conclusively, ML shows promising potential to identify patients at risk prior to surgery, surpassing conventional statistics. Still, larger datasets are needed to achieve higher discrimination performances for large-scale clinical implementation in the future.
BibTeX
@inproceedings{winter2024enhancing,
  author = {Axel Winter and Robin van de Water and Bjarne Pfitzner and Marius Ibach and Christoph Riepe and Robert Ahlborn and Lara Faraj and Felix Krenzien and Eva Maria Dobrindt and Jonas Raakow and Igor M. Sauer and Bert Arnrich and Katharina Beyer and Christian Denecke and Johann Pratschke and Max M. Maurer},
  title = {{Enhancing Preoperative Outcome Prediction: A Comparative Retrospective Case–Control Study on Machine Learning versus the International Esodata Study Group Risk Model for Predicting 90-Day Mortality in Oncologic Esophagectomy}},
  booktitle = {Cancers},
  year = {2024},
  doi = {10.3390/cancers16173000},
}
Figure