Multimodal analysis of whole slide images in colorectal cancer

Jitendra Jonnagaddala, Miljana Shulajkovska, Anton Gradisek, Toni Rose Jue, Qifeng Zhou, Yuzhi Guo, Jamil Mahmoud El Chayeb, Ruijiang Li, Jana Lipková, Jakob Nikolas Kather, Junzhou HuangNpj Digit. Medicine 2025 · DOI

Introduction

Colorectal cancer (CRC) diagnosis has traditionally relied on the histopathological examination of tissue slides—the "gold standard" for clinical decision-making. In the era of digital pathology, these slides are converted into high-resolution Whole Slide Images (WSIs), which artificial intelligence (AI) can analyze to identify patterns beyond human perception. However, a single image is only one facet of a patient's profile. Prognosis and treatment response are equally influenced by genomic mutations, clinical history, and radiological scans.

This paper synthesizes a decade of research into multimodal AI for CRC, exploring how the integration of WSIs with disparate data sources can move diagnostic models toward a more holistic, "multidisciplinary tumor board" approach.

Methodology and Fusion Strategies

The authors conducted a systematic review of 1,601 studies published between 2014 and 2024, identifying 22 high-quality papers that specifically integrated WSIs with other modalities. A core focus of these studies is the technical "fusion" of data, which generally falls into three categories:

  • Early Fusion: Raw data or initial features are combined at the start of the pipeline.
  • Intermediate Fusion: The most prevalent approach, where separate neural networks process each data type into "latent representations" before merging them. This often utilizes attention mechanisms to dynamically weight the importance of different data types.
  • Late Fusion: Independent models generate predictions for each modality, which are then aggregated for a final decision.
Figure 1: Schematic representation of early, intermediate, and late fusion strategies for multimodal data integration.

The Multimodal Landscape

The review highlights a diverse ecosystem of data and computational tools. While The Cancer Genome Atlas (TCGA) remains the primary public resource for paired WSI and genomic data, researchers are increasingly utilizing institutional datasets from global clinical centers.

Figure 2: Taxonomy of datasets, modalities, feature extractors, and downstream tasks in multimodal CRC research.

As shown in the taxonomy above, the field is transitioning from task-specific models to Foundation Models (e.g., BERT for text, ResNet or Vision Transformers for images). These models are pre-trained on massive datasets to understand fundamental medical concepts before being fine-tuned for specific CRC tasks like survival prediction or biomarker detection.

Key Results and Clinical Utility

The integration of multiple data sources consistently outperforms unimodal approaches:

  • Survival Prediction: Models combining WSIs with genomic profiles achieved C-indices of approximately 0.82, significantly surpassing traditional TNM staging or single-modality AI.
  • Biomarker Detection: In predicting Microsatellite Instability (MSI)—a critical indicator for immunotherapy—multimodal models improved accuracy by up to 14% compared to image-only models.
  • Risk Stratification: Multimodal AI demonstrates a superior ability to differentiate between high-risk and low-risk patients, potentially guiding more personalized chemotherapy regimens.

Challenges and Future Directions

Despite significant technical progress, several bottlenecks hinder clinical adoption:

  1. External Validation: Most models are developed and tested on internal datasets, often failing to maintain performance when exposed to different scanners or diverse patient populations.
  2. Temporal Alignment: Clinical data is often collected at different time points (e.g., a biopsy followed weeks later by genomic testing). Developing models that can handle this "time lag" is a critical area for future work.
  3. Interpretability: As fusion architectures become more complex, they often become "black boxes." Enhancing the explainability of these models is essential for gaining clinician trust.
  4. Data Scarcity: High-quality, truly multimodal datasets remain rare and expensive to produce.

Conclusion

Multimodal AI represents the next frontier in precision oncology for colorectal cancer. By shifting from single-source snapshots to an integrated understanding of pathology, genomics, and clinical data, these models offer the potential for unprecedented diagnostic and prognostic accuracy. However, standardized validation and improved interpretability remain necessary milestones before these tools can be integrated into routine clinical workflows.

Summary updated

BibTeX
@article{jonnagaddala2025multimodal,
  author = {Jitendra Jonnagaddala and Miljana Shulajkovska and Anton Gradisek and Toni Rose Jue and Qifeng Zhou and Yuzhi Guo and Jamil Mahmoud El Chayeb and Ruijiang Li and Jana Lipková and Jakob Nikolas Kather and Junzhou Huang},
  title = {{Multimodal analysis of whole slide images in colorectal cancer}},
  journal = {Npj Digit. Medicine},
  year = {2025},
  doi = {10.1038/s41746-025-02095-y},
}
Figure