Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging

Andriy Myronenko, Ziyue Xu, Dong Yang, Holger R. Roth, Daguang Xu

Introduction

In digital pathology, Whole Slide Images (WSIs) present a significant computational challenge due to their immense size, often exceeding billions of pixels. Identifying cancerous regions within these images is a high-stakes task where a few clusters of cells determine a patient's diagnosis. Researchers from NVIDIA have developed a novel Multiple Instance Learning (MIL) framework that moves beyond treating tissue patches as independent entities, instead accounting for the spatial and semantic dependencies crucial for accurate diagnosis.

The MIL Framework and its Limitations

Standard approaches to WSI analysis involve Multiple Instance Learning, where an image is divided into small patches, or "instances," which are collectively treated as a "bag." A single label is typically assigned to the entire bag (e.g., a cancer grade), but the specific patches contributing to that label are unknown.

A major drawback of traditional MIL is the assumption of instance independence. In clinical practice, pathologists rely on the context and relationships between different tissue structures—such as the Gleason patterns in prostate cancer—to make a diagnosis. Treating patches in isolation ignores these vital spatial dependencies.

The process of extracting a 'bag' of patches from a Whole Slide Image (WSI) for Multiple Instance Learning.

Proposed Methodology: Pyramid Transformer MIL

To address these limitations, the authors introduced two key innovations designed to bridge the gap between local patch analysis and global slide context:

1. Transformer-based MIL Architecture

The researchers integrated self-attention Transformer blocks into the MIL pipeline. Unlike standard convolutional neural networks (CNNs) that focus on local features, self-attention allows every patch in a bag to interact with every other patch, capturing global context.

The most advanced version, Pyramid Transformer MIL, embeds transformer blocks at multiple stages of a ResNet50 backbone. This multi-scale approach allows the model to capture dependencies across different levels of semantic abstraction, from low-level textures to high-level tissue structures.

The Pyramid Transformer MIL architecture, showing the integration of Transformer encoder layers with a CNN backbone to capture multi-scale dependencies.

2. Instance-wise Pseudo-labeling

Weak supervision—having only one label for thousands of patches—often limits model performance. The authors developed an instance-wise pseudo-labeling strategy to provide finer-grained supervision. By identifying patches with the highest and lowest attention scores from an ensemble of models, they assigned "pseudo-labels" to the top and bottom 10% of patches. This allowed for the inclusion of a patch-level loss function, guiding the CNN backbone to focus on the most relevant features and reducing overfitting.

Experimental Evaluation

The framework was evaluated using the PANDA (Prostate cANcer graDe Assessment) challenge dataset, which includes over 11,000 WSIs. The task involved predicting ISUP grades (0–5) based on complex Gleason patterns.

The Pyramid Transformer MIL achieved state-of-the-art results:

  • Leaderboard Dominance: A single model achieved a Quadratic Weighted Kappa (QWK) of 0.932 on the hidden test set.
  • Winning Performance: A 10-model ensemble reached a QWK of 0.94136, a score that would have secured first place on the original PANDA Kaggle leaderboard among over 1,000 competing teams.
  • Generalizability: The pseudo-labeling strategy consistently improved the performance of various baseline MIL models by approximately 1%.

Conclusion

By accounting for dependencies between tissue patches, the Pyramid Transformer MIL framework significantly advances the state-of-the-art in digital pathology. The integration of multi-scale self-attention and strategic pseudo-labeling provides a more robust and context-aware approach to Whole Slide Imaging, mirroring the diagnostic process of human pathologists. Future work may explore the transition to pure Vision Transformer architectures to further enhance computational efficiency and performance.

Summary updated

Abstract

Multiple instance learning (MIL) is a key algorithm for classification of whole slide images (WSI). Histology WSIs can have billions of pixels, which create enormous computational and annotation challenges. Typically, such images are divided into a set of patches (a bag of instances), where only bag-level class labels are provided. Deep learning based MIL methods calculate instance features using convolutional neural network (CNN). Our proposed approach is also deep learning based, with the following two contributions: Firstly, we propose to explicitly account for dependencies between instances during training by embedding self-attention Transformer blocks to capture dependencies between instances. For example, a tumor grade may depend on the presence of several particular patterns at different locations in WSI, which requires to account for dependencies between patches. Secondly, we propose an instance-wise loss function based on instance pseudo-labels. We compare the proposed algorithm to multiple baseline methods, evaluate it on the PANDA challenge dataset, the largest publicly available WSI dataset with over 11K images, and demonstrate state-of-the-art results.
BibTeX
@inproceedings{myronenko2021accounting,
  author = {Andriy Myronenko and Ziyue Xu and Dong Yang and Holger R. Roth and Daguang Xu},
  title = {{Accounting for Dependencies in Deep Learning Based Multiple Instance Learning for Whole Slide Imaging}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=bRMKpVN_ckw},
}
Figure