Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer with Deep Learning

Hailing Liu, Yu Zhao, Fan Yang, Xiaoying Lou, Feng Chung Wu, Hang Li, Xiaohan Xing, Tingying Peng, Bjoern Menze, Junzhou Huang, Shujun Zhang, Anjia Han, Jianhua Yao, Xinjuan FanBME 2022 · DOI

TL;DR

This paper introduces a novel deep learning model (MMIL) that integrates histopathology images and serum biomarkers to predict lymph node metastasis (LNM) in colorectal cancer (CRC) patients before surgery. This multimodal approach significantly outperforms traditional imaging methods and offers a more accurate, accessible tool for treatment planning.

What Problem Does This Paper Address?

Accurate preoperative prediction of lymph node metastasis (LNM) is crucial for tailoring treatment strategies in colorectal cancer (CRC). Current methods, like CT and MRI, have limitations in sensitivity and specificity, often failing to detect micrometastases. While genomic tests exist, they can be costly and unstable. This leaves a critical gap in reliably identifying patients who need more aggressive treatment versus those who might benefit from less invasive approaches, impacting patient outcomes and resource allocation.

Methods

The core of this work is the Multimodal Multiple Instance Learning (MMIL) model. It combines two key data streams:

  1. Histopathology Image Analysis: Whole-slide images (WSIs) are processed using a Multiple Instance Learning (MIL) framework. This involves breaking down WSIs into smaller "tiles" (instances), extracting features from these tiles using a pre-trained ResNet-18, selecting the most discriminative features, and then generating a WSI-level representation. An attention-based mechanism is used to weigh the importance of different tiles.
  2. Biomarker Analysis: Serum tumor-specific biomarkers (CEA, CA125, CA19-9, and AFP) are processed through a fully connected feedforward network to generate a molecular-level representation.

These two representations are then fused using a gating-based attention mechanism and tensor fusion to create a final prediction for LNM status. The model is designed to be interpretable, with heatmaps generated to visualize which regions of the WSI contribute most to the prediction.

Datasets

The study utilized a dataset of 1338 colorectal cancer patients.

  • Discovery Cohort: 1128 patients from The Sixth Affiliated Hospital of Sun Yat-sen University.
  • External Validation Cohort: 210 patients from The First Affiliated Hospital of Sun Yat-sen University and The Fourth Hospital of Harbin Medical University.

The data included H&E-stained biopsy slides (WSIs) and serum tumor-specific biomarker levels.

Key Insights and Ideas

  • Multimodality is Key: Combining histopathology images with serum biomarkers significantly boosts predictive performance compared to using either modality alone. This highlights the complementary nature of these data sources for LNM prediction.
  • Deep Learning for Latent Features: The MMIL model automatically learns complex, discriminative features from histopathology images, bypassing the need for manual feature engineering that traditional methods rely on.
  • Interpretability Matters: Visualizing LNM probabilities on WSIs and identifying key histomic features (like cell density and nuclear morphology) provides valuable insights for pathologists, building trust and understanding in the AI's predictions.
  • Stage-Specific Performance: The model shows particularly strong performance in earlier stages (T1, T2), which is critical as treatment decisions for these stages can differ dramatically.

Key Results

The MMIL model demonstrated robust performance:

  • Discovery Cohort: Achieved high Area Under the Curve (AUC) values across different T stages: 0.926 (T1), 0.878 (T2), 0.809 (T3), and 0.857 (T4). For all stages combined (Tall), the AUC was 0.719.
  • External Validation Cohort: Showed good generalization with AUCs of 0.855 (T1), 0.832 (T2), 0.691 (T3), and 0.792 (T4).
  • Comparison: The MMIL model outperformed preoperative radiology-imaging diagnosis and significantly surpassed models using only image or only biomarker data. For example, it achieved higher AUCs than image-only models by 0.076-0.193 and biomarker-only models by 0.037-0.363.

Limitations and Future Work

  • Single Center Training: The initial model was trained on data from a single center. While external validation showed good generalization, training on data from multiple centers could further enhance domain-invariant feature learning.
  • Retrospective Study & Image Quality: The retrospective nature and potential inconsistencies in H&E staining quality across hospitals were noted limitations.
  • Fine-grained Prediction: The current model categorizes patients into LNM+ or LNM-. Future work could extend this to a multi-class classification task to predict specific N substages (N0, N1, N2), which would require a larger dataset.
  • Transfer Learning: The study explored fine-tuning the model on external data, showing slight improvements, suggesting this is a promising avenue for adapting the model to specific center biases.

Summary updated

Abstract

<i>Objective</i>. To develop an artificial intelligence method predicting lymph node metastasis (LNM) for patients with colorectal cancer (CRC). <i>Impact Statement</i>. A novel interpretable multimodal AI-based method to predict LNM for CRC patients by integrating information of pathological images and serum tumor-specific biomarkers. <i>Introduction</i>. Preoperative diagnosis of LNM is essential in treatment planning for CRC patients. Existing radiology imaging and genomic tests approaches are either unreliable or too costly. <i>Methods</i>. A total of 1338 patients were recruited, where 1128 patients from one centre were included as the discovery cohort and 210 patients from other two centres were involved as the external validation cohort. We developed a Multimodal Multiple Instance Learning (MMIL) model to learn latent features from pathological images and then jointly integrated the clinical biomarker features for predicting LNM status. The heatmaps of the obtained MMIL model were generated for model interpretation. <i>Results</i>. The MMIL model outperformed preoperative radiology-imaging diagnosis and yielded high area under the curve (AUCs) of 0.926, 0.878, 0.809, and 0.857 for patients with stage T1, T2, T3, and T4 CRC, on the discovery cohort. On the external cohort, it obtained AUCs of 0.855, 0.832, 0.691, and 0.792, respectively (T1-T4), which indicates its prediction accuracy and potential adaptability among multiple centres. <i>Conclusion</i>. The MMIL model showed the potential in the early diagnosis of LNM by referring to pathological images and tumor-specific biomarkers, which is easily accessed in different institutes. We revealed the histomorphologic features determining the LNM prediction indicating the model ability to learn informative latent features.
BibTeX
@inproceedings{liu2022preoperative,
  author = {Hailing Liu and Yu Zhao and Fan Yang and Xiaoying Lou and Feng Chung Wu and Hang Li and Xiaohan Xing and Tingying Peng and Bjoern Menze and Junzhou Huang and Shujun Zhang and Anjia Han and Jianhua Yao and Xinjuan Fan},
  title = {{Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer with Deep Learning}},
  booktitle = {BME},
  year = {2022},
  doi = {10.34133/2022/9860179},
}
Figure