Abstract
Deep learning methods such as convolutional neural networks (CNNs) are difficult to directly utilize to analyze whole slide images (WSIs) due to the large image dimensions. We overcome this limitation by proposing a novel two-stage approach. First, we extract a set of representative patches (called mosaic) from a WSI. Each patch of a mosaic is encoded to a feature vector using a deep network. The feature extractor model is fine-tuned using hierarchical target labels of WSIs, i.e., anatomic site and primary diagnosis. In the second stage, a set of encoded patch-level features from a WSI is used to compute the primary diagnosis probability through the proposed Pay Attention with Focus scheme, an attention-weighted averaging of predicted probabilities for all patches of a mosaic modulated by a trainable focal factor. Experimental results show that the proposed model can be robust, and effective for the classification of WSIs.
Introduction
In digital pathology, Whole Slide Images (WSIs) present a significant computational challenge due to their gigapixel scale. Standard deep learning models cannot process these images in their entirety. While researchers typically partition WSIs into smaller patches, aggregating these patch-level predictions into a single, accurate diagnosis remains a complex task. This is fundamentally a Multiple Instance Learning (MIL) problem, where a "bag" of instances (patches) is associated with a global label, but the contribution of individual instances is unknown.
The proposed "Pay Attention with Focus" framework addresses this by leveraging hierarchical clinical labels and a novel focal attention mechanism to improve diagnostic accuracy and model interpretability.
Methodology
The proposed pipeline consists of two primary stages:
1. Mosaic Preparation and Hierarchical Fine-tuning
To manage the vast amount of data, a "mosaic" of representative patches is extracted from each WSI using clustering techniques. The feature extractor is then fine-tuned using hierarchical labels—specifically, the anatomic site (e.g., Lung) and the primary diagnosis (e.g., LUAD). By training the model to recognize the relationship between the site and the diagnosis, it learns more biologically relevant features compared to standard flat classification.
2. FocAtt-MIL: The Focal Attention Mechanism
The core of the second stage is the FocAtt-MIL scheme. Unlike traditional attention models that simply weight patches by importance, FocAtt-MIL introduces a trainable Focal Factor ($\gamma$).
As shown in the figure above, the model computes a global "WSI Context" vector. An attention network determines the importance of each patch ($a_i$) based on this context. Simultaneously, a focal network generates the vector $\gamma$, which modulates the patch-level predicted probabilities ($p_i$). The final prediction is an attention-weighted average where probabilities are adjusted by the focal factor, allowing the model to dynamically sharpen its focus on critical diagnostic features.
Experimental Results
The model was evaluated using massive datasets from The Cancer Genome Atlas (TCGA), including a lung cancer dataset (2,580 slides) and a pan-cancer dataset (7,841 slides).
- Lung Cancer Classification: FocAtt-MIL achieved an accuracy of 88% (0.92 AUC) in distinguishing between LUAD and LUSC, significantly outperforming existing benchmarks.
- Ablation Studies: Experiments demonstrated that the inclusion of the WSI Context and the Focal Factor were critical, contributing to 4% and 6% improvements in accuracy, respectively.
- Pan-cancer Robustness: The model maintained high performance across 30 different primary diagnoses, demonstrating its versatility across various anatomic sites. Visualizations of the attention weights confirmed that the model focuses on malignant structures while ignoring healthy tissue, providing a level of transparency crucial for clinical adoption.
Conclusion
The "Pay Attention with Focus" scheme provides a robust framework for gigapixel image classification. By integrating hierarchical clinical knowledge and a novel focal attention mechanism, the model achieves state-of-the-art performance while offering interpretability through attention heatmaps. Future work will focus on refining mosaic selection and testing the model's performance on more heterogeneous, real-world clinical datasets.
Reference: Kalra, S., et al. "Pay Attention with Focus: A Novel Learning Scheme for Classification of Whole Slide Images." Accepted in MICCAI, 2021.
Summary updated