A deep learning based graph-transformer for whole slide image classification

Yi Zheng, Rushin H. Gindra, Margrit Betke, Jennifer Beane, Vijaya B. KolachalamamedRxiv 2021 · DOI

Abstract

Deep learning is a powerful tool for assessing pathology data obtained from digitized biopsy slides. In the context of supervised learning, most methods typically divide a whole slide image (WSI) into patches, aggregate convolutional neural network outcomes on them and estimate overall disease grade. However, patch-based methods introduce label noise in training by assuming that each patch is independent with the same label as the WSI and neglect the important contextual information that is significant in disease grading. Here we present a Graph-Transformer (GT) based framework for processing pathology data, called GTP, that interprets morphological and spatial information at the WSI-level to predict disease grade. To demonstrate the applicability of our approach, we selected 3,024 hematoxylin and eosin WSIs of lung tumors and with normal histology from the Clinical Proteomic Tumor Analysis Consortium, the National Lung Screening Trial, and The Cancer Genome Atlas, and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from those that have normal histology. Our model achieved consistently high performance on binary (tumor versus normal: mean overall accuracy = 0.975 ± 0.013) as well as three-label (normal versus LUAD versus LSCC: mean accuracy = 0.932 ± 0.019) classification on held-out test data, underscoring the power of GT-based deep learning for WSI-level classification. We also introduced a graph-based saliency mapping technique, called GraphCAM, that captures regional as well as contextual information and allows our model to highlight WSI regions that are highly associated with the class label. Taken together, our findings demonstrate GTP as a novel interpretable and effective deep learning framework for WSI-level classification.

Introduction

In digital pathology, a single Whole Slide Image (WSI)—a high-resolution scan of a tissue biopsy—can exceed a gigabyte in size. Because these images are too large for standard deep learning models to process at once, the industry standard is to divide them into thousands of small "patches." However, this approach has two major flaws: it assumes every patch from a "cancerous" slide contains cancer (label noise), and it ignores the spatial relationships between patches (loss of context). Just as a pathologist "zooms out" to see how different tissue structures interact, a model needs to understand the global architecture of the biopsy to make an accurate grade or stage prediction.

Researchers have developed the "Graph-Transformer" (GTP) framework to address these issues. GTP treats massive pathology slides as spatial graphs, combining self-supervised feature extraction with the global reasoning of Transformers. This allows the model to achieve high accuracy in identifying lung cancer subtypes while providing interpretability through heatmaps.

Methods

The GTP framework operates in four distinct stages to process WSI data effectively:

  1. Self-Supervised Feature Extraction: To avoid the need for manual patch-level labels, a SimCLR-style contrastive learning approach was used. A ResNet18 backbone was trained to recognize similar features across 1.8 million patches, creating a robust feature vector for any piece of tissue.
  2. Graph Construction: The WSI is represented as a graph where each patch is a node. Edges are drawn between adjacent patches (8-connectivity), preserving the physical layout of the biopsy.
  3. Graph Convolution & Pooling: A Graph Convolutional (GC) layer aggregates information from neighboring patches to capture local context. To make the data manageable for a Transformer, mincut pooling reduces the thousands of nodes into a smaller, representative set of clusters.
  4. The Transformer: A standard Transformer layer performs self-attention across these clusters, allowing the model to weigh the importance of different regions across the entire slide (global context).
The GTP framework pipeline: from Whole Slide Image to Graph Construction and final classification via a Graph Transformer.

Additionally, the authors introduced GraphCAM, a novel interpretability tool that produces class-specific saliency maps, highlighting which specific nodes in the graph contributed to a specific diagnosis.

Datasets

The study utilized a massive cohort of 3,024 lung WSIs across three primary sources:

  • NLST (National Lung Screening Trial): 1.8 million patches used exclusively for the self-supervised pre-training of the feature extractor.
  • CPTAC (Clinical Proteomic Tumor Analysis Consortium): 2,071 WSIs used for training and 5-fold cross-validation.
  • TCGA (The Cancer Genome Atlas): 288 WSIs used as a completely independent "held-out" test set to evaluate how well the model generalizes to data from different hospitals.

The tasks included binary classification (Tumor vs. Normal) and three-label classification (Normal vs. Adenocarcinoma [LUAD] vs. Squamous Cell Carcinoma [LSCC]).

Key Results

The GTP framework demonstrated state-of-the-art performance across multiple metrics:

  • Binary Accuracy: Achieved 0.975 ± 0.013 accuracy on the CPTAC dataset for distinguishing tumor from normal tissue.
  • Subtype Accuracy: Achieved 0.932 ± 0.019 accuracy for the three-way classification (Normal/LUAD/LSCC).
  • Generalization: On the independent TCGA test set, the model maintained high performance, proving it wasn't just "memorizing" the quirks of one specific lab's scanners.
ROC and Precision-Recall curves for the TCGA dataset, showing high AUC values for Normal, LUAD, and LSCC classification.

Ablation studies showed that removing either the Transformer or the contrastive learning components significantly degraded performance, validating that the fusion of these technologies is essential to the system's success.

Discussion and Conclusion

The GTP framework successfully mimics the "Zoom-In/Zoom-Out" workflow of a human pathologist by using Graph Convolutions for local neighborhoods and Transformers for global dependencies. Unlike standard Vision Transformers (ViT) that treat patches as a simple sequence, the graph-based approach explicitly respects the 2D spatial topology of the tissue, which is more biologically relevant.

While powerful, the framework faces challenges such as high computational intensity during the initial contrastive learning phase. The current model also uses a rigid 8-connectivity graph based on square patches. Future iterations could explore more "biologically congruent" nodes, such as cell clusters or tissue segments. Despite these limitations, GTP represents a significant step forward in interpretable and effective deep learning for WSI-level classification, with potential applications extending to other cancer types and clinical outcome predictions.

Summary updated

Abstract

Abstract Deep learning is a powerful tool for assessing pathology data obtained from digitized biopsy slides. In the context of supervised learning, most methods typically divide a whole slide image (WSI) into patches, aggregate convolutional neural network outcomes on them and estimate overall disease grade. However, patch-based methods introduce label noise in training by assuming that each patch is independent with the same label as the WSI and neglect the important contextual information that is significant in disease grading. Here we present a Graph-Transformer (GT) based framework for processing pathology data, called GTP, that interprets morphological and spatial information at the WSI-level to predict disease grade. To demonstrate the applicability of our approach, we selected 3,024 hematoxylin and eosin WSIs of lung tumors and with normal histology from the Clinical Proteomic Tumor Analysis Consortium, the National Lung Screening Trial, and The Cancer Genome Atlas, and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from those that have normal histology. Our model achieved consistently high performance on binary (tumor versus normal: mean overall accuracy = 0.975 ± 0.013) as well as three-label (normal versus LUAD versus LSCC: mean accuracy = 0.932 ± 0.019) classification on held-out test data, underscoring the power of GT-based deep learning for WSI-level classification. We also introduced a graphbased saliency mapping technique, called GraphCAM, that captures regional as well as contextual information and allows our model to highlight WSI regions that are highly associated with the class label. Taken together, our findings demonstrate GTP as a novel interpretable and effective deep learning framework for WSI-level classification.
BibTeX
@misc{zheng2021deep,
  author = {Yi Zheng and Rushin H. Gindra and Margrit Betke and Jennifer Beane and Vijaya B. Kolachalama},
  title = {{A deep learning based graph-transformer for whole slide image classification}},
  year = {2021},
  doi = {10.1101/2021.10.15.21265060},
  howpublished = {medRxiv},
}
Figure