Abstract
Predicting patient response to $^{177}$Lu-based Peptide Receptor Radionuclide Therapy (PRRT) remains a significant challenge in managing metastatic neuroendocrine tumors (NETs). This study presents a multimodal deep learning framework that integrates 3D PET/CT imaging with routine laboratory biomarkers to predict progression-free survival (PFS). By leveraging a fusion-based architecture and transfer learning, the model achieves superior predictive performance (AUROC 0.72) compared to single-modality approaches. Explainable AI techniques further validate the model's focus on physiologically relevant features, offering a promising tool for personalized treatment planning.
Introduction
Peptide Receptor Radionuclide Therapy (PRRT) is a cornerstone for treating metastatic NETs, yet clinical response is highly heterogeneous. Current prognostic markers, such as the Ki-67 index or Chromogranin A (CgA) levels, often fail to capture the full complexity of disease progression. There is a critical need for integrated models that can synthesize high-dimensional imaging data and tabular clinical biomarkers to provide accurate, individualized risk assessments.
Methodology
The proposed framework utilizes a tripartite architecture to process diverse data streams:
- Laboratory Branch: A multilayer perceptron (MLP) processes four key biomarkers: AST, ALT, GGT, and CgA.
- Imaging Branches: Two parallel 3D Convolutional Neural Networks (3D-CNNs) extract features from pre-therapeutic Somatostatin Receptor (SR)-PET and CT scans.
- Fusion and Prediction: Features from all branches are concatenated and passed through fully connected layers to predict PFS.
A significant technical contribution is the use of transfer learning for the CT branch, initialized with weights from a pretrained 3D segmentation model. This approach compensates for limited sample sizes by providing the model with prior anatomical knowledge.
The study utilized a retrospective cohort of 116 patients from Charité—Universitätsmedizin Berlin. Patients were classified based on a PFS threshold of one year. To ensure transparency, the team employed UMAP for feature visualization and saliency maps to identify influential anatomical regions.
Results and Discussion
The researchers evaluated several input configurations to determine the impact of each modality on predictive accuracy.
Key findings include:
- Multimodal Superiority: The PET-CT-Laboratory fusion model (with pretrained CT) outperformed all other configurations, achieving an AUROC of 0.72 ± 0.01 and an AUPRC of 0.80 ± 0.02. In contrast, unimodal models trained only on PET or CT scans performed poorly, with AUROCs of 0.42 and 0.54, respectively.
- The Role of Pretraining: Initializing the CT branch with pretrained weights was essential for performance gains, highlighting the importance of transfer learning in medical imaging with small datasets.
- Clinical Stratification: Kaplan-Meier analysis demonstrated that the model effectively stratified patients into high- and low-response groups, with median PFS of 17.25 and 12.43 months, respectively ($p = 0.0001$).
- Explainability: Saliency maps revealed that the multimodal model focused on tumor-relevant regions, whereas unimodal PET models were often misled by physiological tracer accumulation in the bladder.
Conclusion
This work demonstrates that fusing 3D imaging with biochemical markers significantly improves the prediction of PRRT outcomes. While limited by a single-center cohort and sample size, the framework provides a foundation for "digital twin" prognostic models. Future work will focus on multi-center validation and the integration of additional modalities like genomics and digital pathology.
Summary updated