Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning

Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, Faisal Mahmood

TL;DR

The Hierarchical Image Pyramid Transformer (HIPT) scales Vision Transformers to gigapixel whole-slide images by leveraging the natural hierarchical structure of tissue. Using multi-stage self-supervised learning, HIPT learns high-resolution representations from the cellular level up to large tissue regions, outperforming state-of-the-art methods in cancer subtyping and survival prediction while demonstrating exceptional data efficiency.

What Problem Does This Paper Address?

In computational pathology, whole-slide images (WSIs) are digital scans of tissue biopsies that are incredibly massive—often reaching sizes up to $150,000 \times 150,000$ pixels. Standard Vision Transformers (ViTs) cannot process these images directly due to the quadratic computational complexity of self-attention.

To circumvent this, current state-of-the-art methods rely on Multiple Instance Learning (MIL). MIL frameworks typically crop the WSI into small, disconnected $256 \times 256$ patches, use a frozen convolutional neural network (often pretrained on natural images like ImageNet) to extract features, and then apply a global pooling operator to make a slide-level prediction. This approach has major limitations: it completely discards the spatial and hierarchical context across larger tissue regions, and it relies on feature extractors that have never seen a cancer cell.

Methods

The authors introduce HIPT, an architecture designed to match the inherent "part-whole" hierarchy of histopathology images. Instead of treating a slide as a flat bag of patches, HIPT models it as a nested sequence of tokens across three distinct scales:

  1. Cell-Level ($\text{ViT}_{256}\text{-}16$): Processes $256 \times 256$ patches broken into $16 \times 16$ pixel tokens to capture individual cell shapes and local structures.
  2. Patch-Level ($\text{ViT}_{4096}\text{-}256$): Aggregates the [CLS] tokens from the cell level across a larger $4096 \times 4096$ region, capturing cellular organization and the local microenvironment.
  3. Region-Level ($\text{ViT}_{\text{WSI}}\text{-}4096$): Aggregates the [CLS] tokens from the patch level across the entire WSI to form a global slide representation.
The HIPT architecture processes whole-slide images through a bottom-up hierarchical aggregation scheme.

To train this massive architecture without pixel-level annotations, the authors use Hierarchical Self-Supervised Learning. They employ the DINO (student-teacher distillation) framework to pretrain the first two stages sequentially. First, $\text{ViT}_{256}\text{-}16$ is trained on millions of patches. Once frozen, its outputs are used to train $\text{ViT}_{4096}\text{-}256$ on larger regions. Finally, the slide-level transformer is finetuned using weak supervision (slide-level labels like patient survival or cancer subtype).

Datasets

HIPT was pretrained on a massive pan-cancer dataset curated from The Cancer Genome Atlas (TCGA), spanning 33 different cancer types. The pretraining corpus included:

  • 10,678 gigapixel whole-slide images
  • 408,218 $4096 \times 4096$ region images
  • 104 million $256 \times 256$ patches

For evaluation, the model was benchmarked on slide-level classification (Breast BRCA, Lung NSCLC, and Kidney RCC subtyping) and survival prediction across multiple cohorts (IDC, CRC, CCRCC, PRCC, LUAD, STAD). It was also evaluated on patch-level datasets including CRC-100K, BCSS, and BreastPathQ.

Key Insights and Ideas

  • Fixed-Scale Inductive Bias: Unlike natural images where objects change size based on camera distance, pathology images have a fixed physical scale (e.g., 20× magnification always equals $\approx 0.5\mu m/\text{pixel}$). HIPT exploits this by enforcing a rigid hierarchical structure rather than forcing the network to learn scale invariance.
  • Emergent Biological Semantic Segmentation: Without any pixel-level labels, the self-supervised attention heads naturally learn to segment distinct biological structures. The cell-level transformer isolates lymphocytes and stroma, while the patch-level transformer delineates macro-scale structures like tumor nests and desmoplastic stroma.
Multi-head self-attention maps show that HIPT naturally isolates fine-grained cellular features and coarse-grained tissue phenotypes without supervision.
  • Pretraining Prevents Overfitting: Clinical datasets are notoriously small (often $<1000$ samples). By pretraining and freezing the intermediate aggregation layers ($\text{ViT}_{4096}\text{-}256$), HIPT drastically reduces the number of trainable parameters during final finetuning, preventing the model from overfitting.

Key Results

  • Cancer Subtyping: HIPT consistently outperformed top MIL methods like CLAM-SB and DS-MIL. On NSCLC subtyping, HIPT achieved an AUC of 0.952 (vs. CLAM-SB's 0.928).
  • Data Efficiency: In a low-data regime using only 25% of the training labels, HIPT maintained robust performance, outperforming baseline models that used 100% of the data in some tasks. For instance, on NSCLC subtyping with 25% data, HIPT achieved an AUC of 0.923, beating all baselines except 100%-trained CLAM.
  • Survival Prediction: HIPT showed massive gains in context-aware survival tasks, achieving a Concordance Index (c-Index) of 0.634 in Invasive Ductal Carcinoma (IDC), compared to just 0.487 for standard Attention-Based MIL (ABMIL).

Limitations and Future Work

  • Staged Training Bottleneck: Due to hardware constraints, HIPT cannot be trained end-to-end; it must be trained in frozen, sequential stages. This prevents the lower layers from adapting to slide-level feedback.
  • Slide-Level Data Scarcity: Pretraining the final slide-level aggregation layer remains difficult because the number of available whole slides (thousands) is orders of magnitude smaller than the number of patches (millions).
  • Demographic Bias: The pretraining data from TCGA predominantly represents patients of European ancestry, meaning future work must validate HIPT on more diverse, independent global cohorts to ensure clinical generalizability.

Summary updated

Abstract

Vision Transformers (ViTs) and their multi-scale and hierarchical variations have been successful at capturing image representations but their use has been generally studied for low-resolution images (e.g. - 256x256, 384384). For gigapixel whole-slide imaging (WSI) in computational pathology, WSIs can be as large as 150000x150000 pixels at 20X magnification and exhibit a hierarchical structure of visual tokens across varying resolutions: from 16x16 images capture spatial patterns among cells, to 4096x4096 images characterizing interactions within the tissue microenvironment. We introduce a new ViT architecture called the Hierarchical Image Pyramid Transformer (HIPT), which leverages the natural hierarchical structure inherent in WSIs using two levels of self-supervised learning to learn high-resolution image representations. HIPT is pretrained across 33 cancer types using 10,678 gigapixel WSIs, 408,218 4096x4096 images, and 104M 256x256 images. We benchmark HIPT representations on 9 slide-level tasks, and demonstrate that: 1) HIPT with hierarchical pretraining outperforms current state-of-the-art methods for cancer subtyping and survival prediction, 2) self-supervised ViTs are able to model important inductive biases about the hierarchical structure of phenotypes in the tumor microenvironment.
BibTeX
@inproceedings{chen2022scaling,
  author = {Richard J. Chen and Chengkuan Chen and Yicong Li and Tiffany Y. Chen and Andrew D. Trister and Rahul G. Krishnan and Faisal Mahmood},
  title = {{Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure