Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer

Sunghoon Joo, Eun Sook Ko, Soonhwan Kwon, Eunjoo Jeon, Hyungsik Jung, Ji‐Yeon Kim, Myung Jin Chung, Young‐Hyuck ImScientific 2021 · DOI

TL;DR

This paper introduces a multimodal deep learning model that combines clinical data and MR images to predict the pathologic complete response (pCR) to neoadjuvant chemotherapy (NAC) in breast cancer patients. The model achieves superior prediction performance by fusing these diverse data sources, offering a promising tool for personalized treatment strategies.

What Problem Does This Paper Address?

Neoadjuvant chemotherapy (NAC) is a crucial treatment for many breast cancer patients, aiming to shrink tumors before surgery and assess treatment sensitivity. A key indicator of NAC success is the pathologic complete response (pCR), meaning no residual invasive cancer is found in the breast and lymph nodes after treatment. Predicting pCR before NAC is vital for tailoring treatment plans, avoiding ineffective therapies, and improving patient outcomes. While clinical factors and MR imaging have shown some predictive power individually, previous studies often used smaller datasets, limited imaging modalities, or focused only on specific tumor regions. This paper addresses the need for a more robust and comprehensive approach by developing a deep learning model that integrates a wider range of clinical information with detailed MR imaging features.

Methods

The core of this study is a multimodal deep learning architecture designed to fuse information from clinical data and MR images. The model utilizes two parallel 3D ResNet-50 networks to extract features from contrast-enhanced T1-weighted (T1W) subtraction MR images and T2-weighted (T2W) MR images, respectively. These MR images were preprocessed, including isotropic spacing and normalization, and then cropped to a 3D form (224x224x64 voxels) for input. Clinical information, which included normalized numeric features (e.g., age, BMI) and one-hot encoded categorical features (e.g., HER2 status), was processed through a separate fully connected (FC) layer. The features extracted from the MR image networks and the clinical information FC layer were then concatenated and fed into a final FC layer with sigmoid activation for binary prediction of pCR. This multimodal fusion approach is a key innovation, aiming to leverage the complementary information from different data types.

Datasets

The study utilized a retrospective cohort of 536 patients with invasive breast cancer who underwent pre-operative NAC at Samsung Medical Center. The dataset included pretreatment MR images (T1W subtraction and T2W) and associated clinical information. The data was randomly split into a training set (n=429) and a validation set (n=107). No new benchmarks were introduced; rather, the study aimed to improve prediction on existing data types.

Key Insights and Ideas

  • Multimodal Fusion is Key: The most significant finding is that combining clinical information with MR image features dramatically improves the prediction of pCR compared to using either data source alone. This highlights the synergistic value of integrating diverse patient data.
  • 3D Whole-Image Analysis Outperforms Cropped Views: Using entire 3D MR images, including areas beyond the immediate lesion (e.g., axilla and chest wall), led to better performance than models trained on cropped, lesion-focused images. This suggests that subtle, potentially overlooked information in surrounding tissues can be crucial for prediction.
  • Deep Learning's Power in Feature Extraction: The study demonstrates deep learning's ability to automatically extract complex, high-dimensional features from MR images, surpassing traditional radiomics approaches in this context by reducing reliance on manual feature engineering and potentially capturing more nuanced patterns.
  • Clinical Factors Still Matter: While MR images are powerful, clinical information remains a significant predictor. The study found that certain clinical factors, like NAC regimen and Ki-67 levels, were particularly important for the model's prediction.

Key Results

The multimodal deep learning model integrating T1W subtraction images, T2W images, and clinical information achieved the highest prediction performance, with an Area Under the Curve (AUC) of 0.888 in the validation set. This significantly outperformed a model using only clinical information (AUC = 0.827, P < 0.05). Models using only MR images also showed predictive capabilities, with T1W subtraction and T2W images yielding AUCs of 0.725 and 0.663, respectively. Combining T1W and T2W images without clinical data improved performance to an AUC of 0.745. The study also noted that using uncropped MR images led to significantly better AUCs compared to cropped images.

Limitations and Future Work

The authors acknowledge several limitations:

  • Interpretability: Understanding how the deep learning model arrives at its predictions remains challenging due to the complex multimodal fusion and 3D CNN architectures. Future work should explore attention mechanisms or other explainability techniques.
  • Retrospective and Single-Center Study: The study was retrospective and conducted at a single institution using MR scanners from a single vendor (Philips). This limits the generalizability of the model. Future research should involve multi-institutional validation with diverse scanner types.
  • Definition of pCR: The study included residual ductal carcinoma in situ (DCIS) as part of pCR. Different definitions of pCR might yield different results, and incorporating functional MR imaging (like diffusion-weighted imaging - DWI) could potentially improve sensitivity.
  • Limited MR Sequences: The study primarily used T1W subtraction and T2W images. Incorporating contrast-enhanced images from multiple time points (for kinetic information) and DWI could further enhance predictive performance.

The authors suggest that with further technological development and validation on larger, diverse datasets, this multimodal deep learning approach could be integrated into computer-aided diagnosis systems for breast cancer, aiding clinicians in making more informed treatment decisions during NAC.

Summary updated

Abstract

The achievement of the pathologic complete response (pCR) has been considered a metric for the success of neoadjuvant chemotherapy (NAC) and a powerful surrogate indicator of the risk of recurrence and long-term survival. This study aimed to develop a multimodal deep learning model that combined clinical information and pretreatment MR images for predicting pCR to NAC in patients with breast cancer. The retrospective study cohort consisted of 536 patients with invasive breast cancer who underwent pre-operative NAC. We developed a deep learning model to fuse high-dimensional MR image features and the clinical information for the pretreatment prediction of pCR to NAC in breast cancer. The proposed deep learning model trained on all datasets as clinical information, T1-weighted subtraction images, and T2-weighted images shows better performance with area under the curve (AUC) of 0.888 as compared to the model using only clinical information (AUC = 0.827, P < 0.05). Our results demonstrate that the multimodal fusion approach using deep learning with both clinical information and MR images achieve higher prediction performance compared to the deep learning model without the fusion approach. Deep learning could integrate pretreatment MR images with clinical information to improve pCR prediction performance.
BibTeX
@inproceedings{joo2021multimodal,
  author = {Sunghoon Joo and Eun Sook Ko and Soonhwan Kwon and Eunjoo Jeon and Hyungsik Jung and Ji‐Yeon Kim and Myung Jin Chung and Young‐Hyuck Im},
  title = {{Multimodal deep learning models for the prediction of pathologic response to neoadjuvant chemotherapy in breast cancer}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-98408-8},
}
Figure