M2Fusion: Bayesian-Based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Prediction

Quan Liu, Jiawen Yao, Lisha Yao, Xin Chen, Jingren Zhou, Le Lu, Ling Zhang, Zaiyi Liu, Yuankai Huo

TL;DR

$M^2$Fusion is a novel Bayesian-based framework that combines pathology whole-slide images (WSI) and 3D radiology CT scans to predict Microsatellite Instability (MSI) in colorectal cancer. By integrating these modalities at both the feature and decision levels, the model achieves a state-of-the-art AUC of 0.8177, significantly outperforming single-modality approaches.

What Problem Does This Paper Address?

Microsatellite Instability (MSI) is a critical biomarker in colorectal cancer (CRC) because it identifies patients who are likely to respond exceptionally well to immunotherapy. However, standard genetic or immunohistochemical testing for MSI is expensive, complex, and not always accessible in clinical practice.

While deep learning has been used to predict MSI directly from pathology slides (WSIs), these models often struggle with the sheer scale of gigapixel images. Meanwhile, radiology (CT) images also contain MSI-related information but are rarely used in tandem with pathology. The core challenge is the heterogeneity gap: how do you effectively fuse 2D gigapixel pathology patches with 3D radiology volumes? Most existing methods use simple "decision-level" fusion (averaging scores), which misses the complex interactions between different types of medical data.

Methods

The authors propose $M^2$Fusion, a multi-level fusion pipeline grounded in Bayesian theory. The architecture consists of several key components:

  1. Dual-Branch Feature Extraction:
    • Pathology Branch: Uses a ResNet-18 encoder to extract features from 224x224 patches cropped from the WSI. These are aggregated into a single patient-level embedding using max pooling.
  2. Radiology Branch: Employs a "2.5D" approach where six essential CT slices (tumor regions and full slices from three directions) are stacked as a six-channel input for a modified ResNet-18.
  3. Bayesian-based Multi-level Fusion: Instead of just picking one fusion method, the authors use Bayes' theorem to combine:
  4. Decision-level fusion: The prior probability based on individual modality performance.
  5. Feature-level fusion: A Transformer-based model that takes both pathology and radiology embeddings (plus a learnable [CLS] token) to discover cross-modal patterns.
  6. Radiology-Guided Training: The model uses a pre-trained, frozen radiology model to "guide" the training of the pathology feature fusion, helping the network focus on the most relevant diagnostic features.

Datasets

The study utilized an in-house dataset of 352 CRC patients from a provincial hospital.

  • Composition: 46 MSI patients and 306 MSS (Microsatellite Stable) patients.
  • Data Pairs: Each patient had both a gigapixel pathology WSI and a 3D venous-phase CT scan.
  • Evaluation: The authors used 5-fold cross-validation to ensure the robustness of the results, maintaining a consistent MSI/MSS ratio across folds.

Key Insights and Ideas

  • Radiology as a "Prior": The most innovative aspect is treating radiology data as a Bayesian prior. Since CT scans capture macroscopic tumor characteristics that pathology might miss, using them to guide the feature-level fusion helps bridge the scale gap between the two modalities.
  • The Power of "Multi-level": The paper demonstrates that combining feature-level (Transformer-based) and decision-level (probabilistic) fusion is more robust than using either alone. It allows the model to capture both fine-grained interactions and high-level consensus.
  • Transformer vs. MLP: In their ablation studies, the authors found that a Vision Transformer (ViT) outperformed a standard Multi-Layer Perceptron (MLP) for feature fusion, likely due to the Transformer's superior ability to model long-range dependencies between the disparate embeddings.

Key Results

The $M^2$Fusion model outperformed all baseline strategies on the AUC (Area Under the Curve) metric:

  • $M^2$Fusion (Proposed): 0.8177
  • Decision-level Fusion: 0.7908
  • Radiology-only (2.5D): 0.7348
  • Pathology-only (ResNet): 0.6847
  • Feature-level Fusion (without guidance): 0.7289

Notably, the multi-level approach provided a significant boost over the best single-modality performance, proving that pathology and radiology provide complementary information for MSI prediction.

Limitations and Future Work

  • Dataset Size and Balance: The dataset is relatively small (352 cases) and highly imbalanced (only 46 MSI cases). While cross-validation helps, testing on larger, multi-center datasets is necessary for clinical validation.
  • In-house Data: The results are based on a single-site dataset. External validation on public datasets (like TCGA) would strengthen the claims of generalizability.
  • Interpretability: While the model performs well, the "black box" nature of the Transformer fusion makes it difficult for clinicians to see exactly which features in the CT or WSI are driving the MSI prediction. Future work could involve attention map visualizations to improve clinical trust.

Summary updated

Abstract

Colorectal cancer (CRC) micro-satellite instability (MSI) prediction on histopathology images is a challenging weakly supervised learning task that involves multi-instance learning on gigapixel images. To date, radiology images have proven to have CRC MSI information and efficient patient imaging techniques. Different data modalities integration offers the opportunity to increase the accuracy and robustness of MSI prediction. Despite the progress in representation learning from the whole slide images (WSI) and exploring the potential of making use of radiology data, CRC MSI prediction remains a challenge to fuse the information from multiple data modalities (e.g., pathology WSI and radiology CT image). In this paper, we propose $M^{2}$Fusion: a Bayesian-based multimodal multi-level fusion pipeline for CRC MSI. The proposed fusion model $M^{2}$Fusion is capable of discovering more novel patterns within and across modalities that are beneficial for predicting MSI than using a single modality alone, as well as other fusion methods. The contribution of the paper is three-fold: (1) $M^{2}$Fusion is the first pipeline of multi-level fusion on pathology WSI and 3D radiology CT image for MSI prediction; (2) CT images are the first time integrated into multimodal fusion for CRC MSI prediction; (3) feature-level fusion strategy is evaluated on both Transformer-based and CNN-based method. Our approach is validated on cross-validation of 352 cases and outperforms either feature-level (0.8177 vs. 0.7908) or decision-level fusion strategy (0.8177 vs. 0.7289) on AUC score.
BibTeX
@inproceedings{liu2023m,
  author = {Quan Liu and Jiawen Yao and Lisha Yao and Xin Chen and Jingren Zhou and Le Lu and Ling Zhang and Zaiyi Liu and Yuankai Huo},
  title = {{M2Fusion: Bayesian-Based Multimodal Multi-level Fusion on Colorectal Cancer Microsatellite Instability Prediction}},
  booktitle = {MTSAIL/LEAF/AI4Treat/MMMI/REMIA @ MICCAI},
  year = {2023},
  doi = {10.1007/978-3-031-47425-5\_12},
}
Figure