Deep learning on CT scans to predict checkpoint inhibitor treatment outcomes in advanced melanoma

Laurens S. ter Maat, Rob A. J. de Mooij, Isabella A.j. van Duin, Joost J.c. Verhoeff, Sjoerd G. Elias, Tim Leiner, Wouter A. C. van Amsterdam, M.f. Troenokarso, E.r.a.n. Arntz, Franchette W.p.j. van den Berkmortel, M J Boers-Sonderen, Martijn F. Boomsma, A.j.m. van den Eertwegh, Jan Willem de Groot, Geke A.p. Hospers, Djura Piersma, Gerard Vreugdenhil, Hans M. Westgeest, Ellen Kapiteijn, G. Ardine de Wit, Willeke A.m. Blokx, P. J. van Diest, P.a. de Jong, Josien P. W. Pluim, Karijn P.m. Suijkerbuijk, Mitko VetamedRxiv 2023 · DOI

Abstract

Checkpoint inhibitor therapy has transformed the treatment landscape for advanced melanoma, yet a significant portion of patients do not respond, facing potential toxicity and high costs without clinical benefit. This study investigates whether deep learning (DL) applied to baseline computed tomography (CT) scans can provide a predictive biomarker for treatment outcomes. Utilizing a large-scale, multi-center dataset of 730 patients, we developed a 3D deep learning model (DLM) and compared its performance against a model of established clinical predictors. While the DLM achieved statistically significant discriminative value (AUROC 0.607), it failed to outperform or improve upon clinical data (AUROC 0.635). Our analysis suggests that the DLM primarily identifies lesion size and organ location—information already captured by clinical variables—highlighting the necessity of benchmarking AI models against existing clinical standards.

Introduction

Immunotherapy, specifically checkpoint inhibitors targeting PD-1 and CTLA-4, offers a chance at long-term survival for patients with unresectable stage III or IV melanoma. However, the response rate remains approximately 50-60%, and the treatment is associated with severe immune-related adverse events and substantial financial costs. Current biomarkers, such as PD-L1 expression or tumor mutational burden, are often insufficient for precise clinical decision-making.

Deep learning has been proposed as a method to uncover "sub-visual" features in medical imaging—a field known as radiomics—that might predict treatment response. This study represents the first large-scale effort to evaluate whether 3D deep learning on baseline CT scans can provide incremental value over standard clinical predictors in advanced melanoma.

Methods

Patient Cohort and Data Acquisition

We retrospectively identified 730 adult patients treated with first-line anti-PD1 ± anti-CTLA4 therapy across ten medical centers in the Netherlands. The dataset included 2,722 individual melanoma lesions. Clinical benefit was defined as stable disease for at least six months or a partial/complete response at any point during follow-up.

Deep Learning Pipeline

Researchers extracted 3D volumes of interest (VOIs) centered on up to five representative lesions per patient from baseline CT scans. Several architectures were evaluated, with a Squeeze-Excitation ResNet50 emerging as the optimal model. This architecture uses attention mechanisms to weight different channels of information, focusing the network on the most relevant spatial features within the 3D tumor volume. To ensure generalizability, the model was validated using a "leave-one-center-out" cross-validation strategy.

Comparative Analysis

The DLM was benchmarked against a clinical model consisting of five established predictors:

  1. Presence of liver metastases
  2. Presence of brain metastases
  3. Lactate dehydrogenase (LDH) levels
  4. Patient performance status
  5. Number of affected organ sites

Results

The rate of clinical benefit in the cohort was 59.6%. The performance metrics for the predictive models were as follows:

  • Deep Learning Model (DLM): Achieved an Area Under the Receiver Operating Characteristic (AUROC) of 0.607 [95% CI 0.565 – 0.648].
  • Clinical Model: Achieved an AUROC of 0.635 [95% CI 0.592 – 0.678].
  • Combined Model (DLM + Clinical): Achieved an AUROC of 0.635 [95% CI 0.595 – 0.676].

Statistical analysis revealed that the DLM's output was significantly correlated with four of the five clinical variables. Crucially, adding the deep learning features to the clinical model provided no improvement in predictive power, indicating that the AI was not capturing unique biological signals beyond what was already known to clinicians.

Discussion

The Proxy Problem

The primary insight of this study is the identification of the "proxy problem" in medical AI. Through interpretability analysis (t-SNE visualizations), the researchers found that the DLM was essentially learning to categorize lesions based on their size and the organ in which they were located. Because the clinical model already accounts for tumor burden (LDH and number of organs) and specific high-risk locations (liver and brain), the DLM’s findings were redundant.

A Reality Check for Radiomics

Many smaller, single-center studies have reported high accuracies for imaging-based AI. However, this study—one of the largest and most rigorous of its kind—suggests a more cautious interpretation. When models are forced to generalize across different hospitals and scanner types, and when they are compared against robust clinical baselines, the "hidden" predictive value of baseline CT imaging appears limited.

Future Directions

While baseline CT scans may have reached a ceiling in predictive utility, other avenues remain promising:

  • Delta-Radiomics: Analyzing changes in tumor texture and volume between baseline and early-treatment scans.
  • Spectral CT: Utilizing multi-energy imaging to gain better insight into tissue composition and vascularization.
  • Body Composition: Investigating systemic factors like sarcopenia (muscle loss) which can be measured on CT and may influence immunotherapy metabolism.

Conclusion

While deep learning can successfully predict immunotherapy outcomes in melanoma, it does so by leveraging the same spatial and anatomical information used in standard clinical practice. This study underscores the critical importance of evaluating new AI tools against existing clinical benchmarks to ensure they provide genuine added value rather than simply replicating known predictors in a "black box" format.

Summary updated

Abstract

Abstract Introduction Checkpoint inhibitor treatment has proven successful for advanced melanoma. However, a significant fraction of patients does not experience benefit from this treatment, that is also associated with potentially severe toxicity and high costs. Previous research has not yet resulted in adequate biomarkers that can predict treatment outcomes. The present work is the first to investigate the value of deep learning on computed tomography (CT) imaging of melanoma lesions for predicting checkpoint inhibitor treatment outcomes in advanced melanoma. Methods Adult patients that were treated with first line anti-PD1 ± anti-CTLA4 therapy for unresectable stage IIIC or stage IV melanoma were retrospectively identified from ten participating centers. Up to five representative lesions were segmented volumetrically on baseline CT; a deep learning model (DLM) was trained on the corresponding volumes to predict clinical benefit, defined as stable disease for a minimum of six months, or response at any time during follow-up. Optimal hyperparameters and model types (Densenet, Efficientnet, Squeeze-Excitation ResNet, ResNeXt) were iteratively explored. The DLM was compared to a model of previously identified clinical predictors (presence of liver and brain metastasis, level of lactate dehydrogenase, performance status and number of affected organs), and a combination model consisting of both clinical predictors and the DLM. Results A total of 730 eligible patients with 2722 lesions were included. Rate of clinical benefit was 59.6%. The selected deep learning model was a Squeeze-Excitation ResNet with random initialization, trained with the Adam optimizer. The DLM reached an area under the receiver operating characteristic (AUROC) of 0.607 [95% CI 0.565 – 0.648]. In comparison, a model of clinical predictors reached an AUROC of 0.635 [95% CI 0.592 – 0.678]. The combination model reached an AUROC of 0.635 [95% CI 0.595 – 0.676]. None of the differences in AUROC were statistically significant. The output of the DLM was significantly correlated with four of the five input variables of the clinical model. Discussion Although the DLM reached a statistically significant discriminative value, it was unable to improve over previously identified clinical predictors. The most likely cause is that the DLM learns to detect a lesion’s size and organ location, which is information that is already present in the clinical model. Given the substantial sample size and extensive hyperparameter optimization, this indicates that the predictive value of CT imaging of lesions for checkpoint inhibitor response in melanoma is likely limited. The present work shows that the assessment over known clinical predictors is an essential step for imaging-based prediction and brings important nuance to the almost exclusively positive findings in this field.
BibTeX
@misc{maat2023deep,
  author = {Laurens S. ter Maat and Rob A. J. de Mooij and Isabella A.J. van Duin and Joost J.C. Verhoeff and Sjoerd G. Elias and Tim Leiner and Wouter A. C. van Amsterdam and M.F. Troenokarso and E.R.A.N. Arntz and Franchette W.P.J. van den Berkmortel and M J Boers-Sonderen and Martijn F. Boomsma and A.J.M. van den Eertwegh and Jan Willem de Groot and Geke A.P. Hospers and Djura Piersma and Gerard Vreugdenhil and Hans M. Westgeest and Ellen Kapiteijn and G. Ardine de Wit and Willeke A.M. Blokx and P. J. van Diest and P.A. de Jong and Josien P. W. Pluim and Karijn P.M. Suijkerbuijk and Mitko Veta},
  title = {{Deep learning on CT scans to predict checkpoint inhibitor treatment outcomes in advanced melanoma}},
  year = {2023},
  doi = {10.1101/2023.07.25.23293133},
  howpublished = {medRxiv},
}
Figure