Multimodal deep learning with routine clinical data for recurrence risk stratification in HR+/HER2- early breast cancer

Xiaoyan Wu, H. Liu, Jiayi Liu, Bingan Mu, Jianfei Li, Siyu Wang, Fengling Li, X. Lu, Hu Chen, Yulan Peng, Yi Chen, Jiancheng Lv, Hong BuResearch 2026 · DOI

TL;DR

This paper introduces the Multimodal Recurrence Risk Prediction (MRRP) model, an AI system that integrates routine clinical data (pathology images, ultrasound, and reports) to predict the long-term recurrence risk of early breast cancer. MRRP achieves superior performance to single-modality approaches, offering a cost-effective and clinically practical tool for personalized treatment and surveillance.

What Problem Does This Paper Address?

Hormone receptor-positive/human epidermal growth factor receptor 2-negative (HR+/HER2-) early breast cancer (EBC) is the most common subtype, but a significant portion of patients still face a long-term risk of recurrence even after standard treatment. Current methods for assessing this risk, such as clinicopathological factors and multigene assays, have limitations: they can be costly, have limited accessibility, and their predictive accuracy is often suboptimal. This leaves a critical need for more precise, accessible, and cost-effective tools to stratify patients by their recurrence risk, enabling better-informed treatment and follow-up decisions.

Methods

The core of this work is the Multimodal Recurrence Risk Prediction (MRRP) model, built upon a hierarchical transformer-based fusion framework. Key methodological innovations include:

  • Multimodal Data Integration: MRRP leverages routinely available clinical data:
  • Pathology: Whole-slide images (WSIs) are processed to extract deep learning features, morphological features (cell geometry, texture), and topological features (spatial relationships between cells).
  • Ultrasound (US): Grayscale US images are used to extract deep features and quantitative radiomics features. US diagnostic reports are processed using a large language model to extract text semantic features.
  • Clinical Data: Structured parameters like age, tumor size, and lymph node status are incorporated.
  • Hierarchical Fusion Framework:
    • Intramodality Fusion: Cross-attention mechanisms are used to dynamically integrate features within each modality (pathology and US separately).
  • Intermodality Fusion: Fused pathology and US features are concatenated.
  • Clinical Information Fusion: A cross-attention mechanism integrates the combined intermodality features with structured clinical information, followed by a residual connection for enhanced representation.
  • Learnable Compensation Mechanism: To address missing modality data, a learnable compensation mechanism is implemented, allowing the model to adapt and maintain robustness even when some data sources are unavailable.
  • Pretrained Encoders: The study systematically evaluated various pretrained encoders (e.g., UNI for WSI, BioGPT for US text) to identify optimal feature extractors for both unimodal and multimodal tasks.

Datasets

The study utilized a curated cohort of 768 HR+/HER2- EBC patients from the West China Hospital (WCH), Sichuan University. This dataset includes:

  • Whole-slide images (WSIs) of H&E-stained pathology slides.
  • Grayscale ultrasound (US) images.
  • US diagnostic reports.
  • Structured clinical parameters.

The dataset was split into training (5:1 ratio) and test sets, with stratification to ensure balanced sample distribution.

Key Insights and Ideas

  • Pathology is Paramount: Ablation studies revealed that pathology features form the core of the model's predictive power. Removing the WSI module led to a substantial drop in performance, highlighting its central role.
  • Synergistic Value of Multimodality: While pathology is key, US and clinical data provide complementary value, significantly improving overall prediction accuracy.
  • Complex Modality Interactions: The optimal feature extraction and fusion strategies for unimodal tasks do not necessarily translate to multimodal settings. Complex interactions require careful experimentation and tailored fusion mechanisms (like cross-attention).
  • Pretrained Encoders Matter, But Context is Key: While specific encoders excel in unimodal tasks, the best multimodal performance might come from different combinations, emphasizing the need to consider inter-modality synergy.
  • Handling Missing Data is Crucial: The learnable compensation mechanism significantly improves model robustness in real-world scenarios where data is often incomplete, making the model more clinically applicable.

Key Results

  • Superior Performance: The MRRP model achieved a C-index of 0.840 on the test set, outperforming single-modality models: WSI-only (0.757), US-only (0.470), and clinical-only (0.596).
  • Long-Term Prediction: The model demonstrated robust time-dependent AUCs exceeding 0.85 at 3, 5, and 7 years.
  • Clinical Utility: Decision-curve analysis showed that MRRP provided a more consistent and broader net benefit across clinically actionable thresholds compared to clinicopathologic models and treat-all/treat-none strategies, particularly at 5 and 7 years.
  • High-Risk Stratification: Patients classified as high-risk by MRRP had a significantly higher hazard ratio for recurrence (HR = 6.06, P < 0.001) and lower 5-year RFS rates (50.0%) compared to the low-risk group (93.7%).
  • Missing Modality Performance: The learnable compensation mechanism improved performance by 9.4% when WSI was missing and 2.5% when US was missing, bridging the gap between complete and missing data scenarios.

Limitations and Future Work

  • Single-Center Retrospective Study: The analysis is from a single center with an internal train-test split. External validation on multicenter prospective cohorts is needed to confirm generalizability.
  • Treatment Confounding: Detailed adjuvant treatment information (regimens, radiotherapy, endocrine duration) was not fully captured and thus not modeled, potentially leaving residual confounding. Future work should incorporate this.
  • Semi-Automated System: The current MRRP relies on manual tumor annotation. Developing robust automatic segmentation and ROI detection would enhance its automation and workflow integration.
  • Molecular Biomarkers: Emerging molecular biomarkers were not included. Future versions could assess their added prognostic value beyond routine clinicopathologic data.
  • US Data Richness: The limited standalone performance of US data might be improved by incorporating richer multiparametric US data and more standardized acquisition protocols.

Summary updated

BibTeX
@inproceedings{wu2026multimodal,
  author = {Xiaoyan Wu and H. Liu and Jiayi Liu and Bingan Mu and Jianfei Li and Siyu Wang and Fengling Li and X. Lu and Hu Chen and Yulan Peng and Yi Chen and Jiancheng Lv and Hong Bu},
  title = {{Multimodal deep learning with routine clinical data for recurrence risk stratification in HR+/HER2- early breast cancer}},
  booktitle = {Research},
  year = {2026},
  doi = {10.34133/research.1136},
}
Figure