Using Multi-Scale Convolutional Neural Network Based on Multi-Instance Learning to Predict the Efficacy of Neoadjuvant Chemoradiotherapy for Rectal Cancer

Dehai Zhang, Yongchun Duan, Jing Guo, Yaowei Wang, Yun Yang, Zhenhui Li, Kelong Wang, Lin Wu, Minghao YuIEEE Journal of Translational Engineering in Health and Medicine 2022 · DOI

Abstract

Our method has outstanding performance and advantages in predicting the efficacy of neoadjuvant chemoradiotherapy for rectal cancer. Clinical and Translational Impact Statement - This study aims to predict the efficacy of neoadjuvant chemoradiotherapy for rectal cancer to assist clinicians quickly diagnose and formulate personalized treatment plans for patients.

Introduction

For patients with locally advanced rectal cancer (LARC), neoadjuvant chemoradiotherapy (NCRT) followed by surgery is the standard of care. However, patient response is highly variable: while some see their tumors vanish (Good Response), others suffer through toxic side effects with little to no tumor regression (Non-Good Response).

Predicting this response early is a critical goal for personalized oncology, but it is technically challenging. Histopathological Whole Slide Images (WSIs) are gigapixel-scale files—far too large for standard neural networks. Furthermore, pathologists usually only provide a single label for the entire slide, while the slide itself contains millions of pixels of both healthy and cancerous tissue. This paper addresses the "weak supervision" problem: how to train an accurate model when you have massive images but very few labels, eliminating the need for time-consuming manual annotations.

Methodology

The authors propose a multi-scale Convolutional Neural Network (CNN) based on Multi-Instance Learning (MIL). In this setup, the whole slide is treated as a "bag" and small patches of the image are treated as "instances."

Architecture of the proposed multi-scale MIL framework with gated attention.

The architecture introduces three novel components:

  1. Gated Attention Weight Normalization: The model uses a multilayer perceptron to assign "attention scores" to different patches, learning which parts of the slide are most predictive. A weight normalization mechanism decouples the length of weight vectors from their direction, significantly speeding up training convergence.
  2. Bilinear Attention Multi-Scale Feature Fusion: Cancer diagnosis requires looking at both individual cells (high magnification) and overall tissue structure (low magnification). The model extracts patches at both 20× and 5× magnification and uses a bilinear transformation to fuse these features, preventing the loss of contextual information.
  3. Weighted Loss Function: Because healthy tissue usually far outweighs cancerous tissue on a slide, the authors designed a loss function that combines cross-entropy with "dual focal loss" to prevent the model from becoming biased toward the majority healthy class.

Experimental Setup and Datasets

The study evaluated the model across three datasets to ensure robustness and generalizability:

  • LARC Dataset: A private dataset of 150 high-resolution WSIs from the Yunnan Cancer Hospital, specifically focused on rectal cancer treatment response.
  • Camelyon16: A public benchmark containing 399 WSIs for breast cancer metastasis detection.
  • MSKCC: A public dataset of 130 WSIs, also used for breast cancer metastasis.

Results and Discussion

The proposed method outperformed several state-of-the-art MIL frameworks, including CLAM and DSMIL. For rectal cancer (LARC), the model achieved an AUC of 0.7589, a significant improvement over single-scale models. On the Camelyon16 dataset, it reached an AUC of 0.9337, and on the MSKCC dataset, it hit 0.9091.

Key insights from the research include:

  • Contextual Fusion: By mathematically combining the global view (5×) with the local view (20×), the model mimics how a human pathologist moves the microscope stage to confirm a diagnosis.
  • Optimization over Architecture: Rather than just building a deeper network, the gated attention mechanism addresses the "vanishing gradient" problem, making the training process more stable and much faster than traditional MIL methods.
  • Weak Supervision Efficacy: The results demonstrate that deep learning models can effectively find "needles in the haystack" (cancerous regions) on their own without pixel-level maps.

Conclusion

While the results are promising, the primary limitation is the sample size of the rectal cancer dataset (150 slides). Larger multi-center studies are needed to validate the model for clinical use. Future work may explore integrating "vision-language" information, allowing the model to learn from pathology reports alongside images. Ultimately, the goal is to move this technology into the clinic to help doctors formulate personalized treatment plans faster.

Summary updated

Abstract

Our method has outstanding performance and advantages in predicting the efficacy of neoadjuvant chemoradiotherapy for rectal cancer. <b><i>Clinical and Translational Impact Statement</i></b> -This study aims to predict the efficacy of neoadjuvant chemoradiotherapy for rectal cancer to assist clinicians quickly diagnose and formulate personalized treatment plans for patients.
BibTeX
@article{zhang2022using,
  author = {Dehai Zhang and Yongchun Duan and Jing Guo and Yaowei Wang and Yun Yang and Zhenhui Li and Kelong Wang and Lin Wu and Minghao Yu},
  title = {{Using Multi-Scale Convolutional Neural Network Based on Multi-Instance Learning to Predict the Efficacy of Neoadjuvant Chemoradiotherapy for Rectal Cancer}},
  journal = {IEEE Journal of Translational Engineering in Health and Medicine},
  year = {2022},
  doi = {10.1109/jtehm.2022.3156851},
}
Figure