Deep Learning on Histopathological Images to Predict Breast Cancer Recurrence Risk and Chemotherapy Benefit

Gil Shamai, Shachar Cohen, Yoav Binenbaum, Edmond Sabo, Alexandra Cretu, Chen Mayer, Iris Barshack, Tal Goldman, Gil Bar‐Sela, António Polónia, Frederick M. Howard, Alexander T. Pearson, Dezheng Huo, Joseph A. Sparano, Ron Kimmel, Dvir AranmedRxiv 2025 · DOI

TL;DR

Researchers have developed a multimodal AI model that estimates breast cancer recurrence risk and chemotherapy benefit using only routine, low-cost histopathology slides and basic clinical data. By leveraging a massive "foundation model" and validating it against the landmark TAILORx randomized trial, the team demonstrated that AI can achieve prognostic accuracy comparable to expensive genomic tests, potentially democratizing precision oncology in resource-limited settings.

What Problem Does This Paper Address?

For patients with HR+/HER2- early-stage breast cancer, the million-dollar question (or more accurately, the $4,000 question) is: Will chemotherapy actually help?

Currently, the gold standard for answering this is genomic testing, specifically the Oncotype DX 21-gene Recurrence Score (RS). While highly effective, these tests are expensive, require specialized labs, and take weeks to return results. Consequently, they are virtually inaccessible in many parts of the world, leaving doctors to rely on "clinical risk" (tumor size and grade), which often leads to over-treatment. This paper aims to bridge that gap by extracting the same "biological signal" from standard Hematoxylin and Eosin (H&E) stained slides—the kind already produced in every pathology lab globally.

Methods

The authors moved beyond simple convolutional neural networks, opting for a state-of-the-art multimodal deep learning architecture:

  1. The Foundation Model: They utilized Prov-GigaPath, a vision transformer pre-trained on a staggering 171,189 whole-slide images (WSIs). This "foundation" allows the model to understand complex tissue morphology without needing manual annotations for every pixel.
  2. Multiple Instance Learning (MIL): Since a pathology slide is a "gigapixel" image, the model uses a LongNet architecture to aggregate features from thousands of small image tiles. This allows it to capture both local cellular details and global architectural patterns across the entire slide.
  3. Multimodal Integration: The image features are combined with clinicopathologic variables (age, tumor size, PR/ER status) using a linear regression layer to produce the final AI-derived risk score.
  4. Zero-Label Calibration: To handle "domain shift" (differences in how slides look across different hospitals), they developed a novel calibration method that requires only an estimate of the local population's mean recurrence score, rather than expensive local genomic data.

Datasets

The study is notable for its scale and the "quality" of its validation data:

  • Primary Development/Validation: Data from the TAILORx randomized controlled trial (n=8,284). Using trial data is crucial because it allows researchers to see how patients actually fared with or without chemotherapy.
  • External Validation: Six independent cohorts (n=5,497) across the globe, including:
  • Three Israeli medical centers (Carmel, Haemek, Sheba).
  • University of Chicago Medical Center (UCMC).
  • The Australian Breast Cancer Tissue Bank (ABCTB).
  • The Cancer Genome Atlas (TCGA).

Key Insights and Ideas

  • Predicting Benefit, Not Just Risk: Most AI pathology papers predict "survival." This paper goes further by showing the AI score can predict chemotherapy benefit. For example, it confirmed that postmenopausal patients classified as "low AI risk" derived no benefit from chemo, mirroring the results of the actual genomic test.
  • The "Grade" Redundancy: Feature importance analysis showed that once the AI looked at the H&E images, the traditional "histological grade" (assigned by human pathologists) became redundant. The AI was essentially "seeing" the grade and more, capturing subtle morphological features humans might miss.
  • Reclassifying the "High Risk": Perhaps the most impactful finding was that the AI reclassified ~30% of postmenopausal patients who were considered "high risk" by traditional clinical criteria as "low risk." In a real-world setting, this could spare thousands of women from the toxic side effects of unnecessary chemotherapy.
  • Local Interpretability: Using heatmaps, the authors showed the model focuses primarily on tumoral regions and areas of inflammation, providing a "sanity check" for clinicians.

Key Results

  • Accuracy: The model achieved an AUC of 0.898 for predicting high genomic risk (RS ≥ 26) in the TAILORx validation set.
  • Correlation: It showed a high correlation (Pearson r = 0.728) with the actual Oncotype DX scores.
  • Prognostic Power: The hazard ratios (HR) for predicting distant recurrence-free interval were nearly identical between the AI score (HR=2.9) and the actual genomic test (HR=2.6).
  • Generalization: Performance remained robust across all six external cohorts, with AUCs ranging from 0.858 to 0.903, proving the model isn't just "memorizing" one hospital's slides.

Limitations and Future Work

  • Retrospective Nature: While validated on trial data, the study was retrospective. Prospective "shadow" studies in real-world clinics are the next logical step.
  • Randomization Constraints: The TAILORx trial only randomized patients with mid-range scores (11-25). Therefore, the model’s ability to predict chemo benefit in very high-risk patients (RS > 26) relies on statistical estimation rather than direct trial evidence.
  • Node-Positive Patients: While the model showed promise in node-positive patients in external cohorts, TAILORx only included node-negative patients. Validation on a trial like RxPONDER (which focused on node-positive cases) would be a major milestone.
  • Infrastructure: While H&E slides are "low cost," the digital pathology pipeline (scanners and GPU servers) still requires an initial investment that some regions may lack.

Summary updated

Abstract

Genomic testing has transformed treatment decisions for hormone receptor-positive, HER2-negative (HR+/HER2-) early breast cancer; however, it remains inaccessible to many patients worldwide due to high costs and logistical barriers. Here, we developed an artificial intelligence (AI) model using a multimodal deep learning approach that estimates Oncotype DX 21-gene recurrence scores (RS) from routine histopathology images and clinicopathologic variables, including age at diagnosis, tumor size, and receptor status. Using a foundation model pre-trained on 171,189 histopathological slides, we fine-tuned and validated our AI model on the TAILORx randomized trial (n=8,284). Among 2,407 patients in the TAILORx validation, the model classifies 45.6% of patients as low-risk, 42.4% as intermediate risk, and 12.0% as high-risk. For predicting high genomic risk disease (RS≥26), occurring in 15.9% in the TAILORx validation set, the model achieves AUC=0.898. Patient stratification by our model shows strong prognostic value across multiple clinical endpoints, including recurrence-free interval, distant recurrence-free interval, and disease-free survival. Importantly, chemotherapy benefit is demonstrated for premenopausal patients classified by our model as high AI risk and chemotherapy benefit is ruled out for postmenopausal patients classified as low AI risk. External validation across six independent cohorts (n=5,497 patients) demonstrates robust generalization of the AI model for prognostication and prediction of RS. Notably, in postmenopausal patients, the AI model reclassifies approximately 30% of clinically high-risk cases, defined by the MINDACT criteria, as low-risk. These findings demonstrate that artificial intelligence applied to standard histopathology can be a valuable tool for chemotherapy decision-making in HR+/HER2- early breast cancer. This approach can help reduce unnecessary chemotherapy and extend precision medicine, particularly in resource-limited settings, where genomic testing is not widely accessible.
BibTeX
@misc{shamai2025deep,
  author = {Gil Shamai and Shachar Cohen and Yoav Binenbaum and Edmond Sabo and Alexandra Cretu and Chen Mayer and Iris Barshack and Tal Goldman and Gil Bar‐Sela and António Polónia and Frederick M. Howard and Alexander T. Pearson and Dezheng Huo and Joseph A. Sparano and Ron Kimmel and Dvir Aran},
  title = {{Deep Learning on Histopathological Images to Predict Breast Cancer Recurrence Risk and Chemotherapy Benefit}},
  year = {2025},
  doi = {10.1101/2025.05.15.25327686},
  howpublished = {medRxiv},
}
Figure