Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning

Bin Li, Yin Li, Kevin W. Eliceiri

Abstract

We address the challenging problem of whole slide image (WSI) classification. WSIs have very high resolutions and usually lack localized annotations. WSI classification can be cast as a multiple instance learning (MIL) problem when only slide-level labels are available. We propose a MIL-based method for WSI classification and tumor detection that does not require localized annotations. Our method has three major components. First, we introduce a novel MIL aggregator that models the relations of the instances in a dual-stream architecture with trainable distance measurement. Second, since WSIs can produce large or unbalanced bags that hinder the training of MIL models, we propose to use self-supervised contrastive learning to extract good representations for MIL and alleviate the issue of prohibitive memory cost for large bags. Third, we adopt a pyramidal fusion mechanism for multiscale WSI features, and further improve the accuracy of classification and localization. Our model is evaluated on two representative WSI datasets. The classification accuracy of our model compares favorably to fully-supervised methods, with less than 2% accuracy gap across datasets. Our results also outperform all previous MIL-based methods. Additional benchmark results on standard MIL datasets further demonstrate the superior performance of our MIL aggregator on general MIL problems.

Introduction: The Challenge of Gigapixel Pathology

In digital pathology, Whole Slide Images (WSIs) are gargantuan—often reaching resolutions of 40,000 x 40,000 pixels. Because a tumor might occupy less than 1% of the total tissue area, identifying it is a significant "needle in a haystack" problem. WSI classification is typically framed as a Multiple Instance Learning (MIL) problem: a slide is treated as a "bag" of thousands of small image patches (instances). If at least one patch is cancerous, the bag is labeled positive.

However, traditional MIL faces two major hurdles:

  1. Labeling Scarcity: It is prohibitively expensive for pathologists to label every single patch; usually, only a single slide-level label is available.
  2. Aggregation Limitations: Simple methods like max-pooling ignore tissue context, while complex attention mechanisms often struggle to converge when positive instances are extremely rare.

Methodology: The DSMIL Framework

The researchers propose DSMIL (Dual-Stream Multiple Instance Learning), a framework designed to bridge the gap between weak and full supervision through three innovative layers.

The DSMIL framework overview, showcasing self-supervised feature extraction (top) and the multiscale MIL aggregator pipeline (bottom).
  • Self-Supervised Feature Extraction: Instead of relying on generic ImageNet pre-training, the model uses SimCLR (a contrastive learning framework) to learn domain-specific representations. By comparing different augmented views of tissue patches, the model learns to identify relevant histological features—like cell nuclei shapes—without manual labels.
  • Pyramidal Fusion: To mimic a pathologist's workflow, DSMIL processes slides at multiple magnifications (e.g., 5x and 20x). Features from these scales are concatenated into a "feature pyramid," allowing the model to integrate global context with local cellular detail.
  • The Dual-Stream Aggregator: This novel aggregator models instance relations through a two-pronged approach that stabilizes the learning process.

The Dual-Stream Aggregator

The aggregator addresses the "anchor" problem in MIL. Most attention mechanisms try to learn weights for all patches simultaneously, which can be noisy. DSMIL anchors its attention to a "critical instance."

Detailed architecture of the Masked Non-Local Block, which implements the dual-stream attention mechanism by comparing instances to a 'critical' anchor.
  1. Stream 1 (Max-Pooling): Identifies the "critical instance"—the patch with the highest probability of being diseased.
  2. Stream 2 (Distance-based Attention): Calculates attention scores for all other patches based on their similarity (distance) to that critical instance. This creates a "soft selection" of patches that likely share the same pathology, providing a more stable and interpretable decision boundary.

Experimental Results

The model was evaluated on several representative datasets:

  • Camelyon16: On this challenging breast cancer metastasis dataset, DSMIL achieved 89.9% accuracy and 0.916 AUC, significantly outperforming previous MIL methods (e.g., ABMIL at 84.5%). This performance is within 2% of fully supervised baselines that require pixel-perfect annotations.
  • TCGA Lung Cancer: The model achieved 95.7% accuracy in distinguishing between Lung Adenocarcinoma (LUAD) and Lung Squamous Cell Carcinoma (LUSC), setting a new state-of-the-art.
  • Localization: Beyond classification, DSMIL generates high-quality heatmaps that accurately localize tumor regions, proving it learns biologically relevant features for the right reasons.

Conclusion and Future Work

DSMIL demonstrates that self-supervised features and a dual-stream aggregation strategy can nearly match the performance of fully supervised models in medical imaging. While powerful, future work may involve incorporating Graph Neural Networks (GNNs) to explicitly model the spatial geometry of tissue architecture and developing automated methods for optimal scale selection in pyramidal fusion.

Summary updated

Abstract

We address the challenging problem of whole slide image (WSI) classification. WSIs have very high resolutions and usually lack localized annotations. WSI classification can be cast as a multiple instance learning (MIL) problem when only slide-level labels are available. We propose a MIL-based method for WSI classification and tumor detection that does not require localized annotations. Our method has three major components. First, we introduce a novel MIL aggregator that models the relations of the instances in a dual-stream architecture with trainable distance measurement. Second, since WSIs can produce large or unbalanced bags that hinder the training of MIL models, we propose to use self-supervised contrastive learning to extract good representations for MIL and alleviate the issue of prohibitive memory cost for large bags. Third, we adopt a pyramidal fusion mechanism for multiscale WSI features, and further improve the accuracy of classification and localization. Our model is evaluated on two representative WSI datasets. The classification accuracy of our model compares favorably to fully-supervised methods, with less than 2% accuracy gap across datasets. Our results also outperform all previous MIL-based methods. Additional benchmark results on standard MIL datasets further demonstrate the superior performance of our MIL aggregator on general MIL problems. GitHub repository: https://github.com/binli123/dsmil-wsi
BibTeX
@inproceedings{li2021dual,
  author = {Bin Li and Yin Li and Kevin W. Eliceiri},
  title = {{Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=XPL2YcVG91X0},
}
Figure