Foundation Model‐Enabled Multimodal Deep Learning for Prognostic Prediction in Colorectal Cancer with Incomplete Modalities: A Multi‐Institutional Retrospective Study

Linhao Qu, Chengsheng Zhang, Yingyong Hou, Feng Tang, Weiqi Sheng, Donghui Huang, Zhijian SongAdvanced 2026 · DOI

TL;DR

This paper introduces FLARE, a novel multimodal deep learning framework that leverages foundation models to predict colorectal cancer prognosis using pathology images, radiology scans, and clinical reports. FLARE excels at handling missing data and achieves superior accuracy compared to existing methods, offering a more personalized approach to cancer treatment.

What Problem Does This Paper Address?

Accurate prognostic prediction is crucial for tailoring colorectal cancer (CRC) treatment and improving patient outcomes. However, current methods often rely on limited, subjective staging systems (like AJCC and TNM) that fail to integrate the wealth of information available from diverse data sources. Pathological images, radiological scans (CT), and clinical text reports all contain vital clues about tumor characteristics, microenvironment, and patient history. The challenge lies in effectively combining these heterogeneous data types, especially when some data modalities are missing for certain patients, which is a common scenario in real-world clinical practice. Existing multimodal approaches struggle with integrating more than two modalities and often assume complete data, limiting their practical applicability.

Methods

FLARE tackles these challenges with a multi-pronged approach:

  • Foundation Models for Feature Extraction: FLARE utilizes pre-trained modality-specific foundation models (FMs) – a pathology vision-language model (PLIP), a radiology model (MedSAM), and a large language model (BioLinkBERT-large) – to efficiently extract rich, relevant features from each data type. This pre-encoding strategy leverages existing knowledge and reduces the burden on the model when dealing with limited multimodal data.
  • Attention-Based Multi-Branch Framework: To capture complex interactions and ensure diverse feature learning, FLARE employs a multi-branch architecture with attention mechanisms. Each branch focuses on a specific modality, and an "exclusive diversity loss function" encourages these branches to learn distinct, prognosis-relevant features.
  • Addressing Missing Modalities: This is a core innovation. FLARE uses:
  • Modality-aware and Missing-aware Prompts: These are special "tokens" embedded with the data that explicitly signal which modalities are present or absent, helping the model adapt.
  • Learnable Pseudo Embeddings: When a modality is missing, FLARE generates a dynamically optimized "pseudo embedding" to fill the gap, compensating for lost information.
  • Modality-Level Data Augmentation: During training, FLARE randomly drops modalities to simulate missing data scenarios, making the model more robust to incomplete inputs during testing.
  • Prognostic Risk Prediction Network: The processed features are fed into a multimodal prognostic risk prediction network (Multi-Modal SurvNet) to estimate patient risk for survival and progression.

Datasets

The study utilized a large, multi-institutional dataset comprising 1,679 colorectal cancer patients. This dataset was collected from four sources:

  • Fudan University Shanghai Cancer Center (FUSCC)
  • Fudan University Zhongshan Hospital (FUZSH)
  • Fudan University Huashan Hospital (FUHSH)
  • The publicly available TCGA-COAD&READ database

The FUSCC cohort was used for training and internal validation (via 5-fold cross-validation), while FUZSH, FUHSH, and TCGA served as independent external validation sets. This multi-center approach is crucial for demonstrating the generalizability of the model.

Key Insights and Ideas

  • Foundation Models are Game-Changers for Multimodal Medical Data: By leveraging pre-trained FMs, FLARE can extract high-quality features from diverse medical data types, significantly simplifying the process and improving performance, especially with limited datasets.
  • Handling Missing Data is Critical for Real-World Applicability: The innovative prompt and pseudo-embedding strategies are key to FLARE's success, allowing it to perform reliably even when patient data is incomplete. This is a major step towards deploying AI in clinical settings.
  • Multimodal Fusion Outperforms Unimodal Approaches: The study clearly demonstrates that combining information from pathology, radiology, and clinical text provides a more comprehensive and accurate picture of prognosis than any single modality alone.
  • Interpretability Matters: FLARE doesn't just provide a prediction; it offers insights into why. Heatmaps highlight important regions in images and text, and analyses of clinical factors, immune cells, and gene pathways reveal the biological underpinnings of its predictions, building trust and aiding clinical understanding.

Key Results

FLARE demonstrated superior prognostic prediction performance across all four cohorts for both Overall Survival (OS) and Progression-Free Survival (PFS).

  • Risk Stratification: FLARE effectively stratified patients into high- and low-risk groups, with significant differences in survival outcomes (p-values consistently < 0.001 for many comparisons).
  • Concordance Index (C-index): FLARE achieved the highest C-index values across all cohorts, outperforming traditional clinical models (Cox regression) and existing multimodal deep learning baselines. For OS, C-index values ranged from 0.730 to 0.812, and for PFS, they ranged from 0.725 to 0.901.
  • Comparison to Baselines: FLARE showed significant improvements over the Cox model (8.3% to 36.4% C-index improvement) and the best deep learning baseline (4.8% to 9.3% improvement).
  • Unimodal vs. Multimodal: Multimodal approaches consistently outperformed unimodal models, with FLARE leading the pack.
  • Interpretability: Analysis revealed that FLARE's predictions align with known prognostic factors (AJCC staging, pTstage, Nstage) and provide insights into immune microenvironment and gene pathway alterations associated with risk.

Limitations and Future Work

The authors acknowledge several limitations:

  • Retrospective Design: While multi-institutional, the study is retrospective. Prospective validation in real-world clinical workflows is the next crucial step.
  • Potential Distribution Biases: Differences in cohort composition and treatment patterns across centers might introduce biases not fully captured.
  • Missing Molecular Data: The dataset did not include raw genomic sequencing data, which could further enhance biological interpretability and predictive accuracy.
  • Colorectal Cancer Specificity: FLARE was developed and validated for CRC. Its utility in other cancer types needs further investigation.

Future work will focus on:

  • Prospective Validation: Conducting larger, prospective studies.
  • Broader Patient Populations: Expanding validation to more diverse patient groups.
  • Integrating More Modalities: Incorporating high-dimensional genomic data.
  • Cross-Cancer Generalization: Evaluating FLARE's performance across different cancer types.
  • Interventional Trials: Designing studies to assess FLARE's impact on clinical decision-making and patient outcomes.

Summary updated

Abstract

Accurate prognostic prediction for colorectal cancer is essential for optimizing personalized treatment strategies and improving patient outcomes. Current unimodal approaches encounter significant limitations in effectively leveraging multimodal data and confront challenges with the issue of missing modalities. A novel multimodal deep learning framework named FLARE, which integrates pathological images, radiological imaging, and clinical text reports, is introduced to provide accurate risk assessments for colorectal cancer survival and progression. FLARE employs foundation models to achieve efficient feature extraction, utilizes an attention-based multi-branch framework to enhance synergy and distinctiveness across modalities, and incorporates a diversity-promoting loss function. To address the issue of incomplete data, FLARE integrates modality and missing-aware prompts, pseudo embeddings, and a modality-level augmentation strategy, thereby effectively mitigating potential performance degradation. The performance of FLARE is retrospectively assessed using a dataset of 1679 colorectal cancer patients from four independent clinical centers. Its superior prognostic capability is demonstrated through Kaplan-Meier analysis and the concordance index. FLARE effectively stratified patients into high- and low-risk groups. It achieved the highest concordance index across all validation cohorts, significantly outperforming traditional clinical models and existing multimodal methods, thereby highlighting its robust generalizability. Interpretability was enhanced by the comprehensive analyses of clinical factors, immune infiltration patterns, and gene pathways, as well as visualizations of feature importance across multiple modalities. In summary, FLARE establishes a comprehensive and robust framework for multimodal deep learning in medical prognostics, providing an advanced Artificial intelligence, Multimodal Deep Learning, Prognosis prediction, colorectal cancer, foundation modeltool for precision cancer prognosis and intelligent diagnosis.
BibTeX
@inproceedings{qu2026foundation,
  author = {Linhao Qu and Chengsheng Zhang and Yingyong Hou and Feng Tang and Weiqi Sheng and Donghui Huang and Zhijian Song},
  title = {{Foundation Model‐Enabled Multimodal Deep Learning for Prognostic Prediction in Colorectal Cancer with Incomplete Modalities: A Multi‐Institutional Retrospective Study}},
  booktitle = {Advanced},
  year = {2026},
  doi = {10.1002/advs.202510931},
}
Figure