RetMIL: Retentive Multiple Instance Learning for Histopathological Whole Slide Image Classification

Hongbo Chu, Qiehe Sun, Jiawen Li, Yuxuan Chen, Lizhong Zhang, Tian Guan, Anjia Han, Yonghong He

TL;DR

RetMIL introduces a "retention mechanism" to Whole Slide Image (WSI) classification, replacing the memory-intensive self-attention found in Transformers. By utilizing a hierarchical structure and linear-complexity retention, RetMIL achieves state-of-the-art accuracy while maintaining nearly constant GPU memory usage and significantly higher throughput than existing models.

What Problem Does This Paper Address?

In computational pathology, Whole Slide Images (WSIs) are "gigapixel" files—massive digital scans of tissue samples. Because these images are too large for standard neural networks, researchers use Multiple Instance Learning (MIL). In MIL, an image is broken into thousands of small patches (instances), which are then aggregated to make a single slide-level diagnosis.

Currently, the gold standard for aggregation is the Transformer. Transformers are excellent at modeling the correlations between different tissue patches, but they have a fatal flaw: quadratic complexity. As the number of patches increases, memory consumption and processing time grow exponentially ($O(N^2)$). For ultra-long sequences typical of large biopsies, Transformer-based MIL models often become too slow or memory-hungry for practical clinical deployment.

Methods

The authors propose RetMIL, which swaps out the standard Transformer backbone for a Retention Mechanism (inspired by the RetNet architecture used in Large Language Models).

The Core Innovation: Retention

Unlike self-attention, which compares every patch to every other patch simultaneously, the retention mechanism uses a linear approach. It incorporates a "relative distance decay matrix," which allows the model to weigh the importance of patches based on their sequence while maintaining a computational complexity that scales linearly ($O(N)$) with the number of patches.

Hierarchical Architecture

RetMIL processes the WSI sequence in two distinct stages:

  1. Local Level: The WSI sequence is divided into smaller subsequences (e.g., 512 patches). These are updated in parallel using a retention mechanism and then condensed into a single "local" feature embedding via attention pooling.
  2. Global Level: These local embeddings are fused into a global sequence. A serial retention mechanism then processes this global context to capture long-range dependencies across the entire slide.
  3. Classification: Finally, a global attention pooling layer produces the final slide-level representation for the classifier.

Datasets

The model was evaluated on three major benchmarks:

  • CAMELYON (16 & 17): A dataset focused on detecting lymph node metastases in breast cancer (binary classification).
  • BRACS: A multi-classification dataset for breast cancer subtyping.
  • LUNG: A binary classification task for non-small cell lung cancer subtypes (LUAD vs. LUSC), using data from the TCGA repository for training and an internal "cooperative hospital" dataset for testing.

Key Insights and Ideas

  • Breaking the Memory Wall: The most striking insight is that RetMIL maintains "almost constant" GPU memory usage regardless of the sequence length. While models like HIPT or HAG-MIL see memory costs skyrocket as slides get larger, RetMIL remains flat and efficient.
  • The Best of Both Worlds: Retention acts as a hybrid between a Transformer and a Recurrent Neural Network (RNN). It offers the parallel training capabilities of Transformers but the efficient, $O(1)$ inference of RNNs.
  • Spatial Awareness via Sequence: By splitting the WSI into subsequences and using a decay-based retention, the model naturally learns to prioritize local clusters of tissue while still "remembering" the broader context of the slide.

Key Results

  • Superior Performance: On the CAMELYON dataset, RetMIL achieved an F1-score of 87.24%, outperforming the popular TransMIL model by over 3%.
  • Speed and Efficiency: RetMIL demonstrated a 1.5x lead in throughput compared to TransMIL.
  • Robustness to Length: In tests involving "ultra-long" sequences (over 15,000 patches), RetMIL's performance remained stable, whereas Transformer-based models often saw performance degradation or reached hardware limits.
  • Interpretability: Heatmap visualizations (Fig. 4) confirmed that the model's attention scores align closely with the cancer regions manually outlined by pathologists.

Limitations and Future Work

While RetMIL represents a significant leap in efficiency, some areas remain for exploration:

  • Subsequence Sensitivity: The paper uses a fixed subsequence length of 512. The impact of varying this "local window" size on different tissue types isn't fully explored.
  • Feature Extraction: Like most MIL methods, RetMIL relies on a frozen, pre-trained feature extractor (ViT-S/16). Future work could investigate if end-to-end training with the retention mechanism provides further gains.
  • Sequence Ordering: WSIs are 2D structures, but MIL treats them as 1D sequences. While the retention mechanism handles sequences well, exploring 2D-aware retention could be a promising next step.

Summary updated

Abstract

Histopathological whole slide image (WSI) analysis with deep learning has become a research focus in computational pathology. The current paradigm is mainly based on multiple instance learning (MIL), in which approaches with Transformer as the backbone are well discussed. These methods convert WSI tasks into sequence tasks by representing patches as tokens in the WSI sequence. However, the feature complexity brought by high heterogeneity and the ultra-long sequences brought by gigapixel size makes Transformer-based MIL suffer from the challenges of high memory consumption, slow inference speed, and lack of performance. To this end, we propose a retentive MIL method called RetMIL, which processes WSI sequences through hierarchical feature propagation structure. At the local level, the WSI sequence is divided into multiple subsequences. Tokens of each subsequence are updated through a parallel linear retention mechanism and aggregated utilizing an attention layer. At the global level, subsequences are fused into a global sequence, then updated through a serial retention mechanism, and finally the slide-level representation is obtained through a global attention pooling. We conduct experiments on two public CAMELYON and BRACS datasets and an public-internal LUNG dataset, confirming that RetMIL not only achieves state-of-the-art performance but also significantly reduces computational overhead. Our code are available at: https://github.com/Hongbo-Chu/RetMIL.
BibTeX
@inproceedings{chu2024retmil,
  author = {Hongbo Chu and Qiehe Sun and Jiawen Li and Yuxuan Chen and Lizhong Zhang and Tian Guan and Anjia Han and Yonghong He},
  title = {{RetMIL: Retentive Multiple Instance Learning for Histopathological Whole Slide Image Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=7JpuAcu7kp},
}

Loading related content...

Figure