TL;DR
$M^2$Fusion is a novel Bayesian-based framework that combines pathology whole-slide images (WSI) and 3D radiology CT scans to predict Microsatellite Instability (MSI) in colorectal cancer. By integrating these modalities at both the feature and decision levels, the model achieves a state-of-the-art AUC of 0.8177, significantly outperforming single-modality approaches.
What Problem Does This Paper Address?
Microsatellite Instability (MSI) is a critical biomarker in colorectal cancer (CRC) because it identifies patients who are likely to respond exceptionally well to immunotherapy. However, standard genetic or immunohistochemical testing for MSI is expensive, complex, and not always accessible in clinical practice.
While deep learning has been used to predict MSI directly from pathology slides (WSIs), these models often struggle with the sheer scale of gigapixel images. Meanwhile, radiology (CT) images also contain MSI-related information but are rarely used in tandem with pathology. The core challenge is the heterogeneity gap: how do you effectively fuse 2D gigapixel pathology patches with 3D radiology volumes? Most existing methods use simple "decision-level" fusion (averaging scores), which misses the complex interactions between different types of medical data.
Methods
The authors propose $M^2$Fusion, a multi-level fusion pipeline grounded in Bayesian theory. The architecture consists of several key components:
- Dual-Branch Feature Extraction:
- Pathology Branch: Uses a ResNet-18 encoder to extract features from 224x224 patches cropped from the WSI. These are aggregated into a single patient-level embedding using max pooling.
- Radiology Branch: Employs a "2.5D" approach where six essential CT slices (tumor regions and full slices from three directions) are stacked as a six-channel input for a modified ResNet-18.
- Bayesian-based Multi-level Fusion: Instead of just picking one fusion method, the authors use Bayes' theorem to combine:
- Decision-level fusion: The prior probability based on individual modality performance.
- Feature-level fusion: A Transformer-based model that takes both pathology and radiology embeddings (plus a learnable [CLS] token) to discover cross-modal patterns.
- Radiology-Guided Training: The model uses a pre-trained, frozen radiology model to "guide" the training of the pathology feature fusion, helping the network focus on the most relevant diagnostic features.
Datasets
The study utilized an in-house dataset of 352 CRC patients from a provincial hospital.
- Composition: 46 MSI patients and 306 MSS (Microsatellite Stable) patients.
- Data Pairs: Each patient had both a gigapixel pathology WSI and a 3D venous-phase CT scan.
- Evaluation: The authors used 5-fold cross-validation to ensure the robustness of the results, maintaining a consistent MSI/MSS ratio across folds.
Key Insights and Ideas
- Radiology as a "Prior": The most innovative aspect is treating radiology data as a Bayesian prior. Since CT scans capture macroscopic tumor characteristics that pathology might miss, using them to guide the feature-level fusion helps bridge the scale gap between the two modalities.
- The Power of "Multi-level": The paper demonstrates that combining feature-level (Transformer-based) and decision-level (probabilistic) fusion is more robust than using either alone. It allows the model to capture both fine-grained interactions and high-level consensus.
- Transformer vs. MLP: In their ablation studies, the authors found that a Vision Transformer (ViT) outperformed a standard Multi-Layer Perceptron (MLP) for feature fusion, likely due to the Transformer's superior ability to model long-range dependencies between the disparate embeddings.
Key Results
The $M^2$Fusion model outperformed all baseline strategies on the AUC (Area Under the Curve) metric:
- $M^2$Fusion (Proposed): 0.8177
- Decision-level Fusion: 0.7908
- Radiology-only (2.5D): 0.7348
- Pathology-only (ResNet): 0.6847
- Feature-level Fusion (without guidance): 0.7289
Notably, the multi-level approach provided a significant boost over the best single-modality performance, proving that pathology and radiology provide complementary information for MSI prediction.
Limitations and Future Work
- Dataset Size and Balance: The dataset is relatively small (352 cases) and highly imbalanced (only 46 MSI cases). While cross-validation helps, testing on larger, multi-center datasets is necessary for clinical validation.
- In-house Data: The results are based on a single-site dataset. External validation on public datasets (like TCGA) would strengthen the claims of generalizability.
- Interpretability: While the model performs well, the "black box" nature of the Transformer fusion makes it difficult for clinicians to see exactly which features in the CT or WSI are driving the MSI prediction. Future work could involve attention map visualizations to improve clinical trust.
Summary updated