Abstract
Checkpoint inhibitor therapy has transformed the treatment landscape for advanced melanoma, yet a significant portion of patients do not respond, facing potential toxicity and high costs without clinical benefit. This study investigates whether deep learning (DL) applied to baseline computed tomography (CT) scans can provide a predictive biomarker for treatment outcomes. Utilizing a large-scale, multi-center dataset of 730 patients, we developed a 3D deep learning model (DLM) and compared its performance against a model of established clinical predictors. While the DLM achieved statistically significant discriminative value (AUROC 0.607), it failed to outperform or improve upon clinical data (AUROC 0.635). Our analysis suggests that the DLM primarily identifies lesion size and organ location—information already captured by clinical variables—highlighting the necessity of benchmarking AI models against existing clinical standards.
Introduction
Immunotherapy, specifically checkpoint inhibitors targeting PD-1 and CTLA-4, offers a chance at long-term survival for patients with unresectable stage III or IV melanoma. However, the response rate remains approximately 50-60%, and the treatment is associated with severe immune-related adverse events and substantial financial costs. Current biomarkers, such as PD-L1 expression or tumor mutational burden, are often insufficient for precise clinical decision-making.
Deep learning has been proposed as a method to uncover "sub-visual" features in medical imaging—a field known as radiomics—that might predict treatment response. This study represents the first large-scale effort to evaluate whether 3D deep learning on baseline CT scans can provide incremental value over standard clinical predictors in advanced melanoma.
Methods
Patient Cohort and Data Acquisition
We retrospectively identified 730 adult patients treated with first-line anti-PD1 ± anti-CTLA4 therapy across ten medical centers in the Netherlands. The dataset included 2,722 individual melanoma lesions. Clinical benefit was defined as stable disease for at least six months or a partial/complete response at any point during follow-up.
Deep Learning Pipeline
Researchers extracted 3D volumes of interest (VOIs) centered on up to five representative lesions per patient from baseline CT scans. Several architectures were evaluated, with a Squeeze-Excitation ResNet50 emerging as the optimal model. This architecture uses attention mechanisms to weight different channels of information, focusing the network on the most relevant spatial features within the 3D tumor volume. To ensure generalizability, the model was validated using a "leave-one-center-out" cross-validation strategy.
Comparative Analysis
The DLM was benchmarked against a clinical model consisting of five established predictors:
- Presence of liver metastases
- Presence of brain metastases
- Lactate dehydrogenase (LDH) levels
- Patient performance status
- Number of affected organ sites
Results
The rate of clinical benefit in the cohort was 59.6%. The performance metrics for the predictive models were as follows:
- Deep Learning Model (DLM): Achieved an Area Under the Receiver Operating Characteristic (AUROC) of 0.607 [95% CI 0.565 – 0.648].
- Clinical Model: Achieved an AUROC of 0.635 [95% CI 0.592 – 0.678].
- Combined Model (DLM + Clinical): Achieved an AUROC of 0.635 [95% CI 0.595 – 0.676].
Statistical analysis revealed that the DLM's output was significantly correlated with four of the five clinical variables. Crucially, adding the deep learning features to the clinical model provided no improvement in predictive power, indicating that the AI was not capturing unique biological signals beyond what was already known to clinicians.
Discussion
The Proxy Problem
The primary insight of this study is the identification of the "proxy problem" in medical AI. Through interpretability analysis (t-SNE visualizations), the researchers found that the DLM was essentially learning to categorize lesions based on their size and the organ in which they were located. Because the clinical model already accounts for tumor burden (LDH and number of organs) and specific high-risk locations (liver and brain), the DLM’s findings were redundant.
A Reality Check for Radiomics
Many smaller, single-center studies have reported high accuracies for imaging-based AI. However, this study—one of the largest and most rigorous of its kind—suggests a more cautious interpretation. When models are forced to generalize across different hospitals and scanner types, and when they are compared against robust clinical baselines, the "hidden" predictive value of baseline CT imaging appears limited.
Future Directions
While baseline CT scans may have reached a ceiling in predictive utility, other avenues remain promising:
- Delta-Radiomics: Analyzing changes in tumor texture and volume between baseline and early-treatment scans.
- Spectral CT: Utilizing multi-energy imaging to gain better insight into tissue composition and vascularization.
- Body Composition: Investigating systemic factors like sarcopenia (muscle loss) which can be measured on CT and may influence immunotherapy metabolism.
Conclusion
While deep learning can successfully predict immunotherapy outcomes in melanoma, it does so by leveraging the same spatial and anatomical information used in standard clinical practice. This study underscores the critical importance of evaluating new AI tools against existing clinical benchmarks to ensure they provide genuine added value rather than simply replicating known predictors in a "black box" format.
Summary updated