PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

Shengyi Hua, Fang Yan, Tianle Shen, Xiaofan Zhang, Zhang, XiaofanMedical Image Analysis 2023

Abstract

Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning (SSL) methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. The experimental results show the superiority of our models over most tasks and the efficacy of the proposed pretext tasks.

Introduction

In digital pathology, Hematoxylin and Eosin (H&E) and Immunohistochemistry (IHC) are the primary staining modalities. H&E provides structural tissue architecture, while IHC highlights specific molecular markers. Despite the abundance of digitized slides, the high cost and expertise required for manual labeling present a significant bottleneck for supervised learning.

Existing foundation models often leverage self-supervised learning techniques optimized for natural images. However, pathology images possess unique characteristics: scale relativity (where cellular meaning depends on tissue context) and stain disparity (where H&E and IHC represent the same structure differently). PathoDuet addresses these challenges by introducing a specialized SSL framework that mimics a pathologist's workflow—navigating across magnifications and correlating structural H&E stains with functional IHC markers.

Methodology

PathoDuet utilizes a Vision Transformer (ViT) backbone enhanced by two novel components designed to bridge the gap between different image scales and staining types.

The Pretext Token

Unlike standard ViT architectures that process image patches in isolation, PathoDuet introduces a learnable "pretext token." This token serves as a specialized carrier for auxiliary information, such as sub-patches or alternative stain styles. This allows a single encoder to capture the intricate relationships between different inputs without the computational overhead of dual-stream models.

Task Raisers

Lightweight modules called "task raisers" utilize the pretext token to execute two domain-specific tasks:

  1. Cross-Scale Positioning: The model learns to localize a high-resolution local patch within a lower-resolution global context. This encourages the model to learn "tissue grammar"—the spatial relationships between structures—rather than just local textures.
  2. Cross-Stain Transferring: Using Adaptive Instance Normalization (AdaIN), the model projects structural information from H&E images into the style of IHC images. This enables the model to interpret IHC markers even when labeled IHC data is scarce by leveraging the abundance of H&E data.

Datasets and Experimental Setup

The framework was pretrained and evaluated using a combination of large-scale public and specialized datasets:

  • H&E Pretraining: Approximately 11,000 diagnostic whole-slide images (WSIs) from The Cancer Genome Atlas (TCGA).
  • IHC Pretraining: Paired H&E and IHC images from the HyReCo and BCI datasets.
  • Evaluation: Downstream tasks included colorectal cancer subtyping (NCT-CRC-HE), breast cancer metastasis detection (CAMELYON16), and IHC marker prediction (PD-L1, CD5, CD10, CD21) using in-house datasets.

Results and Discussion

PathoDuet consistently outperformed or matched top-tier pathology models like CTransPath and RetCCL across a wide range of diagnostic tasks.

Low-Data Efficiency

On the NCT-CRC-HE subtyping task, PathoDuet achieved 94.3% accuracy using only 0.5% of the training data. This highlights the model's ability to learn robust features that generalize well even with minimal supervision, a critical factor in medical domains where labels are scarce.

WSI-Level Classification

In the CAMELYON16 challenge, the model reached an AUC of 0.956, proving its ability to identify tiny metastatic regions across massive whole-slide images. Similar success was seen in lung (NSCLC) and kidney (RCC) cancer classification tasks.

IHC Domain Superiority

PathoDuet showed significant improvements in IHC marker prediction, particularly in out-of-distribution scenarios where the model was tested on data from different hospitals. By using H&E as a structural "Rosetta Stone," the model effectively transferred knowledge to the IHC domain, outperforming models that lacked explicit cross-stain training.

Comparison with Large-Scale Models

Notably, PathoDuet (trained on 11k slides) remained highly competitive with "giant" models like Virchow (trained on 1.5 million slides). This suggests that domain-specific training strategies—focusing on how to train rather than just data volume—can yield superior efficiency.

Conclusion

PathoDuet introduces a novel approach to pathological foundation models by explicitly modeling cross-scale and cross-stain relationships. By incorporating a pretext token and specialized task raisers, the framework captures the spatial and multi-modal context essential for clinical diagnosis. Future work will focus on expanding the cross-stain transfer mechanism to a broader library of clinical markers and scaling the framework to even larger datasets to unlock further emergent diagnostic capabilities.

Summary updated

Abstract

Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. To validate the efficacy of our models, we evaluate the performance over a wide variety of downstream tasks, including patch-level colorectal cancer subtyping and whole slide image (WSI)-level classification in H&E field, together with expression level prediction of IHC marker, tumor identification and slide-level qualitative analysis in IHC field. The experimental results show the superiority of our models over most tasks and the efficacy of proposed pretext tasks. The codes and models are available at https://github.com/openmedlab/PathoDuet.
BibTeX
@article{hua2023pathoduet,
  author = {Shengyi Hua and Fang Yan and Tianle Shen and Xiaofan Zhang and Zhang, Xiaofan},
  title = {{PathoDuet: Foundation Models for Pathological Slide Analysis of H\&E and IHC Stains}},
  journal = {Medical Image Analysis},
  year = {2023},
}
Figure