Deep learning on CT scans to predict checkpoint inhibitor treatment outcomes in advanced melanoma

Laurens S. ter Maat, Rob A. J. de Mooij, Isabella A.j. van Duin, Joost J.c. Verhoeff, Sjoerd G. Elias, Tim Leiner, Wouter A. C. van Amsterdam, Max F. Troenokarso, Eran R. A. N. Arntz, Franchette W.p.j. van den Berkmortel, Marye J. Boers‐Sonderen, Martijn F. Boomsma, Fons J. M. Van den Eertwegh, Jan Willem de Groot, Geke A.p. Hospers, Djura Piersma, Art Vreugdenhil, Hans M. Westgeest, Ellen Kapiteijn, Andrzej Wit, Willeke A.m. Blokx, P. J. van Diest, Pim A. de Jong, Josien P. W. Pluim, Karijn P.m. Suijkerbuijk, Mitko VetaScientific 2024 · DOI

TL;DR

This study explored using deep learning on CT scans to predict how patients with advanced melanoma will respond to checkpoint inhibitor therapy. While the deep learning model showed some predictive ability, it did not outperform existing clinical predictors and couldn't improve upon them when combined. This suggests that current CT imaging alone may not contain sufficient information to reliably predict treatment outcomes in this context.

What Problem Does This Paper Address?

Immune checkpoint inhibitors (ICIs) have revolutionized the treatment of advanced melanoma, significantly improving patient survival. However, a substantial portion of patients do not respond to these therapies, and ICI treatment can also lead to severe, sometimes irreversible, toxicity and high costs. This creates a critical need for accurate biomarkers that can predict treatment response before therapy begins. Such biomarkers would allow clinicians to identify non-responders early, enabling them to switch to alternative treatments and avoid unnecessary costs and side effects. While some clinical factors (like lactate dehydrogenase levels and presence of metastases) are known predictors, they are not sufficiently accurate on their own. This paper investigates whether deep learning applied to baseline CT scans of metastatic lesions can provide a more accurate predictive tool.

Methods

The researchers developed a deep learning model (DLM) to predict clinical benefit from ICI treatment in advanced melanoma patients. The DLM was trained on 3D volumes of metastatic lesions from baseline CT scans. The model's architecture was based on a Squeeze-Excitation ResNet50, trained from scratch without transfer learning. Hyperparameter tuning was performed using a systematic approach, exploring various model architectures, learning rates, and regularization techniques. For prediction, the model was trained to predict lesion-level benefit, and these predictions were then aggregated to a patient-level prediction using a 'minimum' function. The DLM's performance was evaluated using a nested cross-validation approach, including a leave-one-center-out strategy to assess generalizability. Its performance was compared against a previously established clinical model based on known predictors and a combined model incorporating both the DLM and clinical predictors.

Datasets

The study utilized a retrospective dataset of 730 eligible patients with advanced melanoma from ten participating centers. This dataset comprised 2722 metastatic lesions from baseline contrast-enhanced CT scans. The data was collected from prospectively maintained registry data. This multicenter dataset is highlighted as the largest of its kind for this specific application.

Key Insights and Ideas

  • CT Imaging's Limited Predictive Power: The study suggests that baseline CT scans of melanoma lesions, even when analyzed with advanced deep learning, may not contain enough information to reliably predict response to checkpoint inhibitors. This challenges the notion that imaging alone can solve the biomarker problem.
  • Clinical Predictors Remain Crucial: Despite the advancements in deep learning, established clinical predictors (like liver/brain metastases, LDH levels, performance status, and number of affected organs) still hold significant predictive value and are not easily surpassed by imaging-based models.
  • Overlap in Information: The DLM's learned representations showed significant correlation with several clinical predictors. This suggests that the DLM might be learning information that is already implicitly captured by the clinical model, rather than discovering entirely new predictive features from the images.
  • Nuance to Existing Literature: The paper brings a critical perspective to the field, noting that many previous studies reporting positive findings for imaging-based prediction may be overoptimistic due to smaller sample sizes or less rigorous validation. This large-scale, multicenter study provides a more grounded assessment.

Key Results

  • DLM Performance: The deep learning model achieved a leave-one-center-out cross-validated AUROC of 0.607 for predicting clinical benefit.
  • Clinical Model Performance: The established clinical model achieved an AUROC of 0.635.
  • Combined Model Performance: The combination of the DLM and clinical model also achieved an AUROC of 0.635.
  • No Significant Improvement: The differences in AUROC between the DLM, the clinical model, and the combined model were not statistically significant. The DLM did not improve predictive value over the clinical model.
  • Correlation with Clinical Factors: The DLM's output was significantly correlated with four out of the five input variables of the clinical model, indicating an overlap in the information captured.
  • Lesion Size vs. Prediction: At the lesion level, the DLM's prediction of benefit was weakly but significantly negatively correlated with lesion volume, suggesting larger lesions might be associated with lower predicted benefit.

Limitations and Future Work

Limitations: * Exclusion of Patients: A significant number of patients were excluded due to the unavailability of contrast-enhanced baseline CT scans, potentially introducing selection bias. * Limited Number of Lesions: The study limited the analysis to a maximum of five lesions per patient, which might not capture the full predictive information if more lesions were present. However, a sensitivity analysis suggested this was unlikely to significantly alter the conclusions.

Future Work: * Assessing Added Value: The authors emphasize the importance of rigorously assessing the added value of imaging-based models over existing clinical predictors, a step often overlooked. * On-Treatment Scans: While less clinically relevant for early decision-making, on-treatment scans have shown better predictive performance in other studies and could be explored further. * Advanced Imaging Techniques: Future research could investigate spectral CT imaging or body composition metrics extracted from baseline CT. * Other Modalities: Exploring other imaging modalities or a combination of modalities might be necessary to achieve accurate baseline predictions.

Summary updated

Abstract

Immune checkpoint inhibitor (ICI) treatment has proven successful for advanced melanoma, but is associated with potentially severe toxicity and high costs. Accurate biomarkers for response are lacking. The present work is the first to investigate the value of deep learning on CT imaging of metastatic lesions for predicting ICI treatment outcomes in advanced melanoma. Adult patients that were treated with ICI for advanced melanoma were retrospectively identified from ten participating centers. A deep learning model (DLM) was trained on volumes of lesions on baseline CT to predict clinical benefit. The DLM was compared to and combined with a model of known clinical predictors (presence of liver and brain metastasis, level of lactate dehydrogenase, performance status and number of affected organs). A total of 730 eligible patients with 2722 lesions were included. The DLM reached an area under the receiver operating characteristic (AUROC) of 0.607 [95%CI 0.565-0.648]. In comparison, a model of clinical predictors reached an AUROC of 0.635 [95%CI 0.59 -0.678]. The combination model reached an AUROC of 0.635 [95% CI 0.595-0.676]. Differences in AUROC were not statistically significant. The output of the DLM was significantly correlated with four of the five input variables of the clinical model. The DLM reached a statistically significant discriminative value, but was unable to improve over known clinical predictors. The present work shows that the assessment over known clinical predictors is an essential step for imaging-based prediction and brings important nuance to the almost exclusively positive findings in this field.
BibTeX
@inproceedings{maat2024deep,
  author = {Laurens S. ter Maat and Rob A. J. de Mooij and Isabella A.J. van Duin and Joost J.C. Verhoeff and Sjoerd G. Elias and Tim Leiner and Wouter A. C. van Amsterdam and Max F. Troenokarso and Eran R. A. N. Arntz and Franchette W.P.J. van den Berkmortel and Marye J. Boers‐Sonderen and Martijn F. Boomsma and Fons J. M. Van den Eertwegh and Jan Willem de Groot and Geke A.P. Hospers and Djura Piersma and Art Vreugdenhil and Hans M. Westgeest and Ellen Kapiteijn and Andrzej Wit and Willeke A.M. Blokx and P. J. van Diest and Pim A. de Jong and Josien P. W. Pluim and Karijn P.M. Suijkerbuijk and Mitko Veta},
  title = {{Deep learning on CT scans to predict checkpoint inhibitor treatment outcomes in advanced melanoma}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-81188-2},
}
Figure