Deep learning-based image analysis predicts PD-L1 status from H&E-stained histopathology images in breast cancer

Gil Shamai, Amir Livne, António Polónia, Edmond Sabo, Alexandra Cretu, Gil Bar‐Sela, Ron KimmelNature Communications 2022 · DOI

TL;DR

Researchers have developed a deep learning framework capable of predicting PD-L1 biomarker status in breast cancer patients using only standard, low-cost H&E-stained tissue slides. Achieving an AUC of up to 0.93, the system not only matches the predictive power of expensive specialized staining but also acts as a "safety net" to identify human interpretation errors and staining deficiencies in clinical practice.

What Problem Does This Paper Address?

In modern oncology, immunotherapy (specifically PD-1/PD-L1 inhibitors) has revolutionized treatment. However, these drugs only work if the tumor expresses the PD-L1 protein. Currently, determining PD-L1 status requires Immunohistochemistry (IHC)—a specialized staining process that is expensive, time-consuming, and consumes limited patient tissue.

Furthermore, IHC interpretation is notoriously subjective; different pathologists or different antibody kits can yield conflicting results. This paper addresses the need for a more robust, accessible, and objective way to screen for these biomarkers using Hematoxylin and Eosin (H&E) slides, which are already routinely produced for every cancer biopsy worldwide.

Methods

The authors employed a deep learning approach centered on a Convolutional Neural Network (CNN) inspired by the ResNet architecture. Key methodological highlights include:

  • Custom Annotation Tool: To create a high-quality "ground truth," the team developed software that allowed expert pathologists to efficiently label thousands of images by comparing H&E slides directly with their IHC counterparts.
  • Handling Class Imbalance: Since most breast cancer patients are PD-L1 negative, the authors used Focal Loss as their training objective. This forces the model to focus on "hard" examples (the rare positives) rather than being overwhelmed by the easy-to-classify negatives.
  • Transfer Learning for Calibration: To ensure the model worked on data from different hospitals (which may have different staining "styles"), they used a calibration step. By fine-tuning only the last layer of the CNN on a small subset of a new cohort, they significantly improved the model's external validity.
  • Feature Visualization: The team used t-SNE embedding to map the high-dimensional features the AI "sees" into a 2D space, allowing them to verify that the model was picking up on biologically relevant structures like immune cell infiltration and tumor cellularity.

Datasets

The study utilized two major independent cohorts:

  1. BCCA (British Columbia Cancer Agency): A massive dataset of 3,376 patients used for training and primary testing.
  2. MA31: An independent clinical trial cohort of 275 patients used for external validation. The images consisted of Tissue Microarrays (TMAs)—small, circular cores of tumor tissue—rather than full-size "Whole Slide Images."

Key Insights and Ideas

  • Morphology as a Proxy for Molecules: The most profound insight is that a tumor’s "architectural signature" (visible on a standard H&E slide) is deeply linked to its molecular status. High PD-L1 expression correlates with hypercellularity, solid growth patterns, and high immune cell presence—features the CNN learned to recognize.
  • AI as a Quality Assurance Tool: The model was particularly effective at flagging "discordant" cases. In instances where the AI predicted a low score but a pathologist had marked the case as positive, a second review often revealed that the initial IHC stain was deficient or the interpretation was likely an error.
  • Democratizing Biomarker Testing: Because H&E staining is ubiquitous and cheap, this system could allow clinics in resource-limited settings to screen patients for immunotherapy eligibility without needing expensive IHC infrastructure.

Key Results

  • High Accuracy: The system achieved an Area Under the Curve (AUC) of 0.91–0.93 on the BCCA cohort for PD-L1 prediction.
  • External Validation: On the independent MA31 clinical trial cohort, the model achieved an AUC of 0.854, which improved to 0.886 after the simple calibration step.
  • Clinical Utility: By setting a specific "low-prediction" threshold, the system could potentially spare 70% of patients from needing IHC testing while maintaining 100% sensitivity for identifying PD-L1 positive cases in the MA31 cohort.
  • PD-1 Prediction: The model also successfully predicted PD-1 status (another related biomarker) with an AUC of 0.848, despite the data being extremely imbalanced (only 3% positive).

Limitations and Future Work

  • TMA vs. WSI: The study was conducted on Tissue Microarrays (TMAs). While effective, TMAs only represent a tiny fraction of a tumor. Future work needs to transition these models to Whole Slide Images (WSIs) to account for tumor heterogeneity.
  • Interpretability: While t-SNE and Grad-CAM provide some clues, deep learning models remain somewhat "black boxes." More work is needed to translate the AI's "features" into specific histological terms that pathologists can easily verify.
  • Prospective Validation: The study was retrospective. The next step is a prospective clinical trial to see how the AI performs as a real-time decision support tool in a live hospital environment.

Summary updated

Abstract

Programmed death ligand-1 (PD-L1) has been recently adopted for breast cancer as a predictive biomarker for immunotherapies. The cost, time, and variability of PD-L1 quantification by immunohistochemistry (IHC) are a challenge. In contrast, hematoxylin and eosin (H&E) is a robust staining used routinely for cancer diagnosis. Here, we show that PD-L1 expression can be predicted from H&E-stained images by employing state-of-the-art deep learning techniques. With the help of two expert pathologists and a designed annotation software, we construct a dataset to assess the feasibility of PD-L1 prediction from H&E in breast cancer. In a cohort of 3,376 patients, our system predicts the PD-L1 status in a high area under the curve (AUC) of 0.91 - 0.93. Our system is validated on two external datasets, including an independent clinical trial cohort, showing consistent prediction performance. Furthermore, the proposed system predicts which cases are prone to pathologists miss-interpretation, showing it can serve as a decision support and quality assurance system in clinical practice.
BibTeX
@inproceedings{shamai2022deep,
  author = {Gil Shamai and Amir Livne and António Polónia and Edmond Sabo and Alexandra Cretu and Gil Bar‐Sela and Ron Kimmel},
  title = {{Deep learning-based image analysis predicts PD-L1 status from H\&E-stained histopathology images in breast cancer}},
  booktitle = {Nature Communications},
  year = {2022},
  doi = {10.1038/s41467-022-34275-9},
}
Figure