Multimodal histopathologic models stratify hormone receptor-positive early breast cancer

Kevin Boehm, Omar S. M. El Nahhas, Antonio Marra, Pier Selenica, Hannah Y. Wen, Britta Weigelt, Evan D. Paul, Pavol Čekan, Ramona Erber, Chiara Maria Lavinia Loeffler, Elena Guerini‐Rocco, Nicola Fusco, Chiara Frascarelli, Eltjona Mane, Elisabetta Munzone, Silvia Dellapasqua, Paola Zagami, Giuseppe Curigliano, Pedram Razavi, Jorge S. Reis‐Filho, Fresia Pareja, Sarat Chandarlapaty, Sohrab P. Shah, Jakob Nikolas KatherbioRxiv 2024 · DOI

TL;DR

This paper introduces "Orpheus," a multimodal AI model that predicts breast cancer recurrence risk using standard H&E stained pathology slides and accompanying text reports. Orpheus achieves performance comparable to or exceeding current gold-standard multigene assays, offering a more accessible and cost-effective alternative for guiding adjuvant chemotherapy decisions globally.

What Problem Does This Paper Address?

For hormone receptor-positive, HER2-negative early breast cancer, a critical challenge is identifying patients who will truly benefit from adjuvant chemotherapy. The Oncotype DX Recurrence Score (RS) is a validated multigene assay that helps make this decision, but its high cost and long turnaround time limit its widespread adoption, especially outside of the US. This creates a need for a more accessible, faster, and cost-effective method to stratify patients by recurrence risk.

Methods

The researchers developed "Orpheus," a multimodal transformer model. It integrates two types of data:

  • Whole-slide images (WSIs): Standard H&E stained pathology slides are processed using a vision transformer.
  • Synoptic text reports: Pathology reports are processed using a language transformer.

These two modalities are then fused using tensor fusion to create a comprehensive prediction. The model was trained and validated on a large dataset of over 6,200 patients across three independent cohorts.

Datasets

The study utilized data from three independent cohorts comprising a total of 6,203 patients with HR+/HER2- early breast cancer.

  • MSK-BRCA cohort: 5,176 patients for training, validation, and testing. This cohort also included genomic data for a subset.
  • IEO-BRCA cohort: 452 patients for external validation.
  • MDX-BRCA cohort: 575 patients for external validation.

Key Insights and Ideas

  • Histopathology as a rich data source: Standard H&E slides, often overlooked for complex predictive tasks, contain significant information for predicting recurrence risk.
  • Multimodality matters: Combining image and text data significantly improves prediction accuracy compared to using either modality alone.
  • AI can democratize advanced diagnostics: Orpheus offers a potential pathway to bring sophisticated risk stratification to regions where expensive multigene assays are not feasible.
  • Uncovering biological underpinnings: The model's attention mechanisms revealed that features like tumor cell size heterogeneity, immune cell infiltration, and stromal fraction are key drivers of high-risk predictions, offering new biological insights.

Key Results

  • Accurate RS prediction: The multimodal model achieved a Pearson correlation of r = 0.68 (95% C.I. 0.64-0.73) for predicting the Oncotype DX recurrence score.
  • High-risk classification: For predicting high-risk disease (RS > 25), the model achieved an AUROC of 0.89 (95% C.I. 0.83-0.94), outperforming a traditional nomogram (AUROC 0.49).
  • Generalizability: The model performed well on external international cohorts, demonstrating robustness.
  • Clinical utility: At an operating point of 94.4% precision and 33.3% recall, Orpheus could potentially identify a significant portion of patients who might benefit from adjuvant chemotherapy without the need for further testing.
  • Cost-effectiveness: The AI-driven approach is estimated to save approximately $1,271 per patient compared to the standard laboratory assay.

Limitations and Future Work

  • Ground truth reliance: The model's "ground truth" is the Oncotype DX RS, meaning it cannot inherently discriminate recurrence risk better than the assay itself. Future work will involve using clinical outcomes (e.g., distal recurrence) as the ground truth for time-to-event modeling.
  • Interpretability: While attention mechanisms provide some insight, deep learning models, including transformers, still suffer from a general lack of true interpretability ("black box" nature). The authors acknowledge that post-hoc explainability is used.
  • Digital pathology infrastructure: Widespread adoption requires institutions to have fully digital pathology workflows, which is not yet universal.
  • Regulatory approval: Clinical deployment will require proper regulatory approval.

The authors suggest that the Orpheus architecture can be tailored for rapid biomarker inference in other tumor types, and future work will explore its application to clinical outcomes and other emerging uses.

Summary updated

Abstract

Abstract For patients with hormone receptor-positive, early breast cancer without HER2 amplification, multigene expression assays including Oncotype DX ® recurrence score (RS) have been clinically validated to identify patients who stand to derive added benefit from adjuvant cytotoxic chemotherapy. However, cost and turnaround time have limited its global adoption despite recommendation by practice guidelines. We investigated if routinely available hematoxylin and eosin (H&E)-stained pathology slides could act as a surrogate triaging data substrate by predicting RS using machine learning methods. We trained and validated a multimodal transformer model, Orpheus, using 6,203 patients across three independent cohorts, taking both H&E images and their corresponding synoptic text reports as input. We showed accurate inference of recurrence score from whole-slide images (r = 0.63 (95% C.I. 0.58 - 0.68); n = 1,029), the raw text of their corresponding reports (r = 0.58 (95% C.I. 0.51 - 0.64); n = 972), and their combination (r = 0.68 (95% C.I. 0.64 - 0.73); n = 964) as measured by Pearson’s correlation. To predict high-risk disease (RS>25), our model achieved an area under the receiver operating characteristic curve (AUROC) of 0.89 (95% C.I. 0.83 - 0.94), and area under the precision recall curve (AUPRC) of 0.64 (95% C.I. 0.60 - 0.82), compared to 0.49 (95% C.I. 0.36 - 0.64) for an existing nomogram based on clinical and pathologic features. Moreover, our model generalizes well to external international cohorts, effectively identifying recurrence risk (r = 0.61, p < 10 -4 , n = 452; r = 0.60, p < 10 -4 , n = 575) and high-risk status (AUROC = 0.80, p < 10 -4 , AUPRC = 0.68, p < 10 -4 , n = 452; AUROC = 0.83, p < 10 -4 , AUPRC = 0.73, p < 10 -4 , n = 575) from whole-slide images. Probing the biologic underpinnings of the model decisions uncovered tumor cell size heterogeneity, immune cell infiltration, a proliferative transcription program, and stromal fraction as correlates of higher-risk predictions. We conclude that at an operating point of 94.4% precision and 33.3% recall, this model could help increase global adoption and shorten lag between resection and adjuvant therapy.
BibTeX
@misc{boehm2024multimodal,
  author = {Kevin Boehm and Omar S. M. El Nahhas and Antonio Marra and Pier Selenica and Hannah Y. Wen and Britta Weigelt and Evan D. Paul and Pavol Čekan and Ramona Erber and Chiara Maria Lavinia Loeffler and Elena Guerini‐Rocco and Nicola Fusco and Chiara Frascarelli and Eltjona Mane and Elisabetta Munzone and Silvia Dellapasqua and Paola Zagami and Giuseppe Curigliano and Pedram Razavi and Jorge S. Reis‐Filho and Fresia Pareja and Sarat Chandarlapaty and Sohrab P. Shah and Jakob Nikolas Kather},
  title = {{Multimodal histopathologic models stratify hormone receptor-positive early breast cancer}},
  year = {2024},
  doi = {10.1101/2024.02.23.581806},
  howpublished = {bioRxiv},
}
Figure