Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapy

Simon Baur, Tristan Ruhwedel, Ekin Böke, Zuzanna Kobus, Gergana Lishkova, Christoph Wetz, Holger Amthauer, Christoph Roderburg, Frank Tacke, Julian M. Rogasch, Wojciech Samek, Henning Jann, Jackie Ma, Johannes EschricharXiv 2025 · DOI · arXiv

Abstract

Predicting patient response to $^{177}$Lu-based Peptide Receptor Radionuclide Therapy (PRRT) remains a significant challenge in managing metastatic neuroendocrine tumors (NETs). This study presents a multimodal deep learning framework that integrates 3D PET/CT imaging with routine laboratory biomarkers to predict progression-free survival (PFS). By leveraging a fusion-based architecture and transfer learning, the model achieves superior predictive performance (AUROC 0.72) compared to single-modality approaches. Explainable AI techniques further validate the model's focus on physiologically relevant features, offering a promising tool for personalized treatment planning.

Introduction

Peptide Receptor Radionuclide Therapy (PRRT) is a cornerstone for treating metastatic NETs, yet clinical response is highly heterogeneous. Current prognostic markers, such as the Ki-67 index or Chromogranin A (CgA) levels, often fail to capture the full complexity of disease progression. There is a critical need for integrated models that can synthesize high-dimensional imaging data and tabular clinical biomarkers to provide accurate, individualized risk assessments.

Methodology

The proposed framework utilizes a tripartite architecture to process diverse data streams:

  1. Laboratory Branch: A multilayer perceptron (MLP) processes four key biomarkers: AST, ALT, GGT, and CgA.
  2. Imaging Branches: Two parallel 3D Convolutional Neural Networks (3D-CNNs) extract features from pre-therapeutic Somatostatin Receptor (SR)-PET and CT scans.
  3. Fusion and Prediction: Features from all branches are concatenated and passed through fully connected layers to predict PFS.

A significant technical contribution is the use of transfer learning for the CT branch, initialized with weights from a pretrained 3D segmentation model. This approach compensates for limited sample sizes by providing the model with prior anatomical knowledge.

Figure 1: Overview of the multimodal deep learning architecture, including input branches, fusion layer, and explainability outputs.

The study utilized a retrospective cohort of 116 patients from Charité—Universitätsmedizin Berlin. Patients were classified based on a PFS threshold of one year. To ensure transparency, the team employed UMAP for feature visualization and saliency maps to identify influential anatomical regions.

Results and Discussion

The researchers evaluated several input configurations to determine the impact of each modality on predictive accuracy.

Figure 2: Summary of the different input setups evaluated in the study, ranging from single-modality to full multimodal fusion.

Key findings include:

  • Multimodal Superiority: The PET-CT-Laboratory fusion model (with pretrained CT) outperformed all other configurations, achieving an AUROC of 0.72 ± 0.01 and an AUPRC of 0.80 ± 0.02. In contrast, unimodal models trained only on PET or CT scans performed poorly, with AUROCs of 0.42 and 0.54, respectively.
  • The Role of Pretraining: Initializing the CT branch with pretrained weights was essential for performance gains, highlighting the importance of transfer learning in medical imaging with small datasets.
  • Clinical Stratification: Kaplan-Meier analysis demonstrated that the model effectively stratified patients into high- and low-response groups, with median PFS of 17.25 and 12.43 months, respectively ($p = 0.0001$).
  • Explainability: Saliency maps revealed that the multimodal model focused on tumor-relevant regions, whereas unimodal PET models were often misled by physiological tracer accumulation in the bladder.

Conclusion

This work demonstrates that fusing 3D imaging with biochemical markers significantly improves the prediction of PRRT outcomes. While limited by a single-center cohort and sample size, the framework provides a foundation for "digital twin" prognostic models. Future work will focus on multi-center validation and the integration of additional modalities like genomics and digital pathology.

Summary updated

Abstract

Peptide receptor radionuclide therapy (PRRT) is an established treatment for metastatic neuroendocrine tumors (NETs), yet long-term disease control occurs only in a subset of patients. Predicting progression-free survival (PFS) could support individualized treatment planning. This study evaluates laboratory, imaging, and multimodal deep learning models for PFS prediction in PRRT-treated patients. In this retrospective, single-center study 116 patients with metastatic NETs undergoing 177Lu-DOTATOC were included. Clinical characteristics, laboratory values, and pretherapeutic somatostatin receptor positron emission tomography/computed tomographies (SR-PET/CT) were collected. Seven models were trained to classify low- vs. high-PFS groups, including unimodal (laboratory, SR-PET, or CT) and multimodal fusion approaches. Explainability was evaluated by feature importance analysis and gradient maps. Forty-two patients (36%) had short PFS (< 1 year), 74 patients long PFS (>1 year). Groups were similar in most characteristics, except for higher baseline chromogranin A (p = 0.003), elevated gamma-GT (p = 0.002), and fewer PRRT cycles (p < 0.001) in short-PFS patients. The Random Forest model trained only on laboratory biomarkers reached an AUROC of 0.59 +- 0.02. Unimodal three-dimensional convolutional neural networks using SR-PET or CT performed worse (AUROC 0.42 +- 0.03 and 0.54 +- 0.01, respectively). A multimodal fusion model laboratory values, SR-PET, and CT -augmented with a pretrained CT branch - achieved the best results (AUROC 0.72 +- 0.01, AUPRC 0.80 +- 0.01). Multimodal deep learning combining SR-PET, CT, and laboratory biomarkers outperformed unimodal approaches for PFS prediction after PRRT. Upon external validation, such models may support risk-adapted follow-up strategies.
BibTeX
@misc{baur2025multimodal,
  author = {Simon Baur and Tristan Ruhwedel and Ekin Böke and Zuzanna Kobus and Gergana Lishkova and Christoph Wetz and Holger Amthauer and Christoph Roderburg and Frank Tacke and Julian M. Rogasch and Wojciech Samek and Henning Jann and Jackie Ma and Johannes Eschrich},
  title = {{Multimodal Deep Learning for Prediction of Progression-Free Survival in Patients with Neuroendocrine Tumors Undergoing 177Lu-based Peptide Receptor Radionuclide Therapy}},
  year = {2025},
  eprint = {2511.05169},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.05169},
}
Figure