90-day mortality prediction in elective visceral surgery using machine learning: a retrospective multicenter development, validation, and comparison study

Christoph Riepe, Robin van de Water, Axel Winter, Bjarne Pfitzner, Lara Faraj, Robert Ahlborn, Maximilian Schulze, Daniela Zuluaga, Christian Schineis, Katharina Beyer, Johann Pratschke, Bert Arnrich, Igor M. Sauer, Max M. MaurerInternational Journal of Surgery 2025 · DOI

TL;DR

This study demonstrates that machine learning (ML) models can effectively predict 90-day mortality in patients undergoing major elective visceral surgery, outperforming traditional risk scores. By training models on aggregated data from multiple organ systems, the researchers achieved more robust and generalizable predictions than with organ-specific models, paving the way for improved preoperative risk stratification.

What Problem Does This Paper Address?

Predicting patient outcomes after major surgery is crucial for informed decision-making, resource allocation, and ultimately, patient care. Visceral surgery, involving organs like the esophagus, stomach, intestines, liver, and pancreas, carries significant risks, including mortality. While traditional scoring systems like the ASA score and Charlson Comorbidity Index exist, they often rely on linear relationships and may not capture the complex interplay of factors influencing surgical outcomes. Machine learning (ML) offers a promising avenue for more nuanced risk prediction, but its application in visceral surgery faces challenges: highly imbalanced data (low mortality rates), the need for extensive external validation to prevent overfitting, and the difficulty of gathering sufficient data for specific organ systems. This paper aims to address these limitations by developing, validating, and comparing ML models for 90-day mortality (90DM) prediction in a large, multi-center cohort of visceral surgery patients, exploring both organ-specific and aggregated approaches.

Methods

The researchers employed a retrospective cohort study design, pooling data from two tertiary university-affiliated surgical centers. They utilized a multi-stage approach:

  1. Data Extraction and Preparation: Electronic health records were used to extract patient characteristics, pre-existing conditions, intervention details, and laboratory values. Data underwent rigorous preprocessing, including exclusion of cases with incomplete endpoint data or features with low completeness, and imputation of missing values using a k-Nearest Neighbors imputer.
  2. Primary ML Analysis: An eXtreme Gradient Boosting (XGBoost) classifier was chosen as the primary ML model due to its strong performance in similar tasks. The models were trained and internally validated using a repeated nested cross-validation setup on data from Center 1. Subsequently, the best-performing models were externally validated on data from Center 2.
  3. Conventional Scoring Comparison: The performance of the ML models was benchmarked against the American Society of Anesthesiologists (ASA) score and the revised Charlson Comorbidity Index (rCCI).
  4. Subgroup Analysis: To assess the impact of organ specificity, individual ML models were developed for esophageal, gastric, intestinal, liver, and pancreatic surgery. These were compared to a combined multi-organ model and a weighted average of the organ-specific models.
  5. Feature Importance: SHapley Additive exPlanations (SHAP) values were used to identify the most influential features contributing to the ML model's predictions.

Datasets

The study utilized a retrospective dataset of 7,711 patients who underwent major elective visceral surgery between January 2014 and December 2022, sourced from two independent, high-volume, tertiary university-affiliated surgical centers. Center 1 served as the internal training cohort (5,813 cases), and Center 2 as the external validation cohort (1,898 cases). A total of 94 preoperatively available features were initially considered, reduced to 78 after feature exclusion.

Key Insights and Ideas

  • ML Outperforms Traditional Scores: Machine learning models, particularly XGBoost, significantly outperform established risk stratification tools like the ASA score and rCCI in predicting 90-day mortality. This suggests ML can capture more complex predictive patterns.
  • Aggregated Models Enhance Robustness: Training ML models on a combined dataset encompassing multiple organ systems leads to better performance and, crucially, lower variability (standard deviation) compared to models trained solely on individual organ systems. This indicates that larger, more diverse training sets can improve generalizability.
  • Key Predictors Identified: The rCCI, patient age, and C-reactive protein (CRP) emerged as the most influential features for predicting 90-day mortality. This aligns with clinical intuition but also highlights the predictive power of readily available clinical data.
  • Organ-Specific Performance Varies: While some organ-specific models (e.g., gastric surgery) showed excellent performance, others (e.g., pancreatic surgery) lagged. This variability underscores the challenge of building robust models for rare events within specific surgical domains.
  • 90-Day Mortality as a Better Benchmark: The study's focus on 90-day mortality, rather than the more traditional 30-day metric, is highlighted as a more relevant and increasingly recognized benchmark for perioperative outcomes.

Key Results

  • The XGBoost classifier achieved an AUROC of 0.86 (±0.01) and an AUPRC of 0.20 (±0.04) for 90-day mortality during external validation.
  • These ML models significantly outperformed the ASA score (AUROC: 0.72, AUPRC: 0.08) and the rCCI (AUROC: 0.81, AUPRC: 0.11).
  • The aggregated multi-organ approach yielded an AUROC of 0.84 (±0.04) and an AUPRC of 0.21 (±0.06) for 90-day mortality, which was superior to the weighted average of organ-specific models (AUROC: 0.82, AUPRC: 0.20).
  • The most influential features identified by SHAP analysis for 90DM prediction were rCCI (0.17), patient age (0.14), and CRP (0.14).
  • Organ-specific models showed varied performance, with gastric surgery achieving the highest AUROC (0.88 ±0.13) and pancreatic surgery the lowest (0.66 ±0.08) for 90DM.

Limitations and Future Work

The authors acknowledge several limitations:

  • Retrospective Design: The retrospective nature of the study may introduce biases, although efforts were made to mitigate these through automated data extraction and strict inclusion criteria.
  • Follow-up Data: Patients with uncomplicated courses might be less likely to have complete 90-day follow-up data, potentially underestimating the true mortality rate.
  • Missing ASA Values: Not all patients had available ASA scores, necessitating imputation, which may not perfectly reflect physician assessments.
  • Socioeconomic Factors: The study did not differentiate based on socioeconomic subgroups, though generalized public health insurance in Germany may mitigate this.

Future work suggested includes:

  • Prospective Studies: The authors emphasize the need for prospective studies to confirm the potential of ML in surgical outcome prediction.
  • Transfer Learning and Federated Learning: For future development, concepts like transfer learning (fine-tuning models trained on large datasets for specific organ systems) and federated learning (enabling collaborative model training across institutions while preserving data privacy) are proposed to overcome data limitations and enhance model robustness.
  • Clinical Integration: Further research is needed to integrate these ML models into clinical practice, addressing surgeon confidence and transparency concerns.

Summary updated

Abstract

ML offers robust preoperative risk stratification for 90DM in elective visceral surgery. Leveraging training across multi-organ cohorts may improve accuracy and robustness compared to organ-specific models. Prospective studies are needed to confirm the potential of ML in surgical outcome prediction.
BibTeX
@article{riepe2025day,
  author = {Christoph Riepe and Robin van de Water and Axel Winter and Bjarne Pfitzner and Lara Faraj and Robert Ahlborn and Maximilian Schulze and Daniela Zuluaga and Christian Schineis and Katharina Beyer and Johann Pratschke and Bert Arnrich and Igor M. Sauer and Max M. Maurer},
  title = {{90-day mortality prediction in elective visceral surgery using machine learning: a retrospective multicenter development, validation, and comparison study}},
  journal = {International Journal of Surgery},
  year = {2025},
  doi = {10.1097/js9.0000000000002372},
}
Figure