TL;DR
This paper introduces the Multimodal Recurrence Risk Prediction (MRRP) model, an AI system that integrates routine clinical data (pathology images, ultrasound, and reports) to predict the long-term recurrence risk of early breast cancer. MRRP achieves superior performance to single-modality approaches, offering a cost-effective and clinically practical tool for personalized treatment and surveillance.
What Problem Does This Paper Address?
Hormone receptor-positive/human epidermal growth factor receptor 2-negative (HR+/HER2-) early breast cancer (EBC) is the most common subtype, but a significant portion of patients still face a long-term risk of recurrence even after standard treatment. Current methods for assessing this risk, such as clinicopathological factors and multigene assays, have limitations: they can be costly, have limited accessibility, and their predictive accuracy is often suboptimal. This leaves a critical need for more precise, accessible, and cost-effective tools to stratify patients by their recurrence risk, enabling better-informed treatment and follow-up decisions.
Methods
The core of this work is the Multimodal Recurrence Risk Prediction (MRRP) model, built upon a hierarchical transformer-based fusion framework. Key methodological innovations include:
- Multimodal Data Integration: MRRP leverages routinely available clinical data:
- Pathology: Whole-slide images (WSIs) are processed to extract deep learning features, morphological features (cell geometry, texture), and topological features (spatial relationships between cells).
- Ultrasound (US): Grayscale US images are used to extract deep features and quantitative radiomics features. US diagnostic reports are processed using a large language model to extract text semantic features.
- Clinical Data: Structured parameters like age, tumor size, and lymph node status are incorporated.
- Hierarchical Fusion Framework:
- Intramodality Fusion: Cross-attention mechanisms are used to dynamically integrate features within each modality (pathology and US separately).
- Intermodality Fusion: Fused pathology and US features are concatenated.
- Clinical Information Fusion: A cross-attention mechanism integrates the combined intermodality features with structured clinical information, followed by a residual connection for enhanced representation.
- Learnable Compensation Mechanism: To address missing modality data, a learnable compensation mechanism is implemented, allowing the model to adapt and maintain robustness even when some data sources are unavailable.
- Pretrained Encoders: The study systematically evaluated various pretrained encoders (e.g., UNI for WSI, BioGPT for US text) to identify optimal feature extractors for both unimodal and multimodal tasks.
Datasets
The study utilized a curated cohort of 768 HR+/HER2- EBC patients from the West China Hospital (WCH), Sichuan University. This dataset includes:
- Whole-slide images (WSIs) of H&E-stained pathology slides.
- Grayscale ultrasound (US) images.
- US diagnostic reports.
- Structured clinical parameters.
The dataset was split into training (5:1 ratio) and test sets, with stratification to ensure balanced sample distribution.
Key Insights and Ideas
- Pathology is Paramount: Ablation studies revealed that pathology features form the core of the model's predictive power. Removing the WSI module led to a substantial drop in performance, highlighting its central role.
- Synergistic Value of Multimodality: While pathology is key, US and clinical data provide complementary value, significantly improving overall prediction accuracy.
- Complex Modality Interactions: The optimal feature extraction and fusion strategies for unimodal tasks do not necessarily translate to multimodal settings. Complex interactions require careful experimentation and tailored fusion mechanisms (like cross-attention).
- Pretrained Encoders Matter, But Context is Key: While specific encoders excel in unimodal tasks, the best multimodal performance might come from different combinations, emphasizing the need to consider inter-modality synergy.
- Handling Missing Data is Crucial: The learnable compensation mechanism significantly improves model robustness in real-world scenarios where data is often incomplete, making the model more clinically applicable.
Key Results
- Superior Performance: The MRRP model achieved a C-index of 0.840 on the test set, outperforming single-modality models: WSI-only (0.757), US-only (0.470), and clinical-only (0.596).
- Long-Term Prediction: The model demonstrated robust time-dependent AUCs exceeding 0.85 at 3, 5, and 7 years.
- Clinical Utility: Decision-curve analysis showed that MRRP provided a more consistent and broader net benefit across clinically actionable thresholds compared to clinicopathologic models and treat-all/treat-none strategies, particularly at 5 and 7 years.
- High-Risk Stratification: Patients classified as high-risk by MRRP had a significantly higher hazard ratio for recurrence (HR = 6.06, P < 0.001) and lower 5-year RFS rates (50.0%) compared to the low-risk group (93.7%).
- Missing Modality Performance: The learnable compensation mechanism improved performance by 9.4% when WSI was missing and 2.5% when US was missing, bridging the gap between complete and missing data scenarios.
Limitations and Future Work
- Single-Center Retrospective Study: The analysis is from a single center with an internal train-test split. External validation on multicenter prospective cohorts is needed to confirm generalizability.
- Treatment Confounding: Detailed adjuvant treatment information (regimens, radiotherapy, endocrine duration) was not fully captured and thus not modeled, potentially leaving residual confounding. Future work should incorporate this.
- Semi-Automated System: The current MRRP relies on manual tumor annotation. Developing robust automatic segmentation and ROI detection would enhance its automation and workflow integration.
- Molecular Biomarkers: Emerging molecular biomarkers were not included. Future versions could assess their added prognostic value beyond routine clinicopathologic data.
- US Data Richness: The limited standalone performance of US data might be improved by incorporating richer multiparametric US data and more standardized acquisition protocols.
Summary updated