Pay Attention with Focus: A Novel Learning Scheme for Classification of Whole Slide Images

Shivam Kalra, Mohammed Adnan, Sobhan Hemati, Taher Dehkharghanian, Shahryar Rahnamayan, Hamid R. Tizhoosh

In the world of digital pathology, a single "Whole Slide Image" (WSI) is a gigapixel-sized puzzle. These images are so massive that standard deep learning models—the kind that recognize cats or cars—simply cannot process them all at once. Researchers usually chop these images into tiny patches, but this creates a new problem: how do you combine thousands of patch-level predictions into one accurate diagnosis for the whole patient?

A new paper titled "Pay Attention with Focus" proposes an elegant two-stage solution that uses hierarchical labels and a novel "focal" attention mechanism to bridge the gap between tiny patches and massive slides.

TL;DR

The authors introduce FocAtt-MIL, a Multiple Instance Learning (MIL) scheme that classifies Whole Slide Images by extracting representative "mosaics" and processing them through an attention mechanism modulated by a trainable "focal factor." By leveraging hierarchical clinical labels (like organ site vs. specific cancer type), the model achieves state-of-the-art accuracy on complex lung cancer and pan-cancer datasets.

What Problem Does This Paper Address?

Standard Convolutional Neural Networks (CNNs) are memory-hungry. Loading a full WSI into a GPU is impossible. The common workaround is to analyze the slide patch-by-patch. However, most datasets only provide a "global" label (e.g., "This slide shows Lung Adenocarcinoma") rather than labeling every individual patch.

This is a classic Multiple Instance Learning (MIL) problem: you have a "bag" of instances (patches), and you know the label of the bag, but not necessarily which specific instances are responsible for that label. Existing MIL methods often struggle with "noise"—patches that contain healthy tissue, blood, or background—which can dilute the final diagnostic signal.

Methods

The researchers propose a two-stage pipeline:

  1. Mosaic Preparation & Hierarchical Fine-tuning: Instead of using every single patch, they extract a "mosaic" of representative patches using clustering. Crucially, they fine-tune their feature extractor (the part of the AI that "sees" the patches) using hierarchical labels. They don't just teach the model to recognize a cancer; they teach it the relationship between the anatomic site (e.g., Lung) and the primary diagnosis (e.g., LUAD). This "context-aware" training helps the model extract more relevant features.

  2. FocAtt-MIL (The "Focus" Mechanism): This is the core innovation. Most attention models simply weight patches by importance. FocAtt-MIL adds a Focal Factor ($\gamma$).

  3. The model computes a global "WSI Context" vector.

  4. An Attention Module decides which patches are important based on this context.
  5. A Focal Network generates a vector ($\gamma$) that modulates the predicted probabilities.
  6. The final prediction is an attention-weighted average where the patch-level probabilities are raised to the power of the focal factor. This allows the model to "sharpen" or "soften" its focus on specific diagnostic features across the entire slide.

Datasets

The study utilized massive datasets from The Cancer Genome Atlas (TCGA):

  • LUAD vs. LUSC: 2,580 slides of Lung Adenocarcinoma and Lung Squamous Cell Carcinoma (approx. 2 TB of data).
  • Pan-cancer Analysis: A larger set of 7,841 slides covering 24 anatomic sites and 30 different primary diagnoses.

Key Insights and Ideas

  • Hierarchy is a Cheat Code: By predicting the conditional probability of a diagnosis given the anatomic site ($P(diagnosis | site)$), the model learns a more structured and biological representation of the data than if it treated every cancer type as an independent category.
  • Global Context Matters for Local Decisions: The "Focal Factor" isn't just a static weight; it’s derived from the entire bag of patches. This means the model's interpretation of a single patch can change based on what it sees elsewhere on the slide.
  • Interpretability via Heatmaps: The attention mechanism isn't just for performance; it acts as a guide. When visualized, the model consistently "attends" to malignant glandular structures and ignores healthy tissue or fibrosis, providing a level of transparency crucial for clinical adoption.

Key Results

  • Lung Cancer Classification: FocAtt-MIL achieved 88% accuracy (0.92 AUC) in distinguishing between LUAD and LUSC, outperforming several established benchmarks in the literature (which hovered around 75-85%).
  • Ablation Success: Removing the "WSI Context" led to a 4% drop in accuracy, and removing the "Focal Factor" led to a 6% drop, proving that these novel components are doing the heavy lifting.
  • Pan-cancer Versatility: The model proved robust across 30 different cancer types. Interestingly, even using a generic feature extractor (DenseNet), the FocAtt-MIL scheme made it competitive with highly specialized pathology models like KimiaNet.

Limitations and Future Work

While the paper demonstrates strong results, a few areas remain open for exploration:

  • Mosaic Selection: The model relies on a specific patch-selection strategy (mosaics). If the initial selection misses a tiny but critical region of cancer, the model cannot recover that information.
  • Computational Complexity: While more efficient than processing every patch, the two-stage process (clustering, then MIL) still requires significant pre-processing.
  • Real-world Variability: The study uses TCGA data, which is high-quality. Future work could test how "Focus" handles the "messier" slides found in smaller, community hospitals with different staining consistencies.

Reference: Kalra, S., et al. "Pay Attention with Focus: A Novel Learning Scheme for Classification of Whole Slide Images." Accepted in MICCAI, 2021.

Summary updated

Abstract

Deep learning methods such as convolutional neural networks (CNNs) are difficult to directly utilize to analyze whole slide images (WSIs) due to the large image dimensions. We overcome this limitation by proposing a novel two-stage approach. First, we extract a set of representative patches (called mosaic) from a WSI. Each patch of a mosaic is encoded to a feature vector using a deep network. The feature extractor model is fine-tuned using hierarchical target labels of WSIs, i.e., anatomic site and primary diagnosis. In the second stage, a set of encoded patch-level features from a WSI is used to compute the primary diagnosis probability through the proposed Pay Attention with Focus scheme, an attention-weighted averaging of predicted probabilities for all patches of a mosaic modulated by a trainable focal factor. Experimental results show that the proposed model can be robust, and effective for the classification of WSIs.
BibTeX
@inproceedings{kalra2021pay,
  author = {Shivam Kalra and Mohammed Adnan and Sobhan Hemati and Taher Dehkharghanian and Shahryar Rahnamayan and Hamid R. Tizhoosh},
  title = {{Pay Attention with Focus: A Novel Learning Scheme for Classification of Whole Slide Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=nRsG9Tk_B6x},
}

Loading related content...

Figure