TL;DR
RetMIL introduces a "retention mechanism" to Whole Slide Image (WSI) classification, replacing the memory-intensive self-attention found in Transformers. By utilizing a hierarchical structure and linear-complexity retention, RetMIL achieves state-of-the-art accuracy while maintaining nearly constant GPU memory usage and significantly higher throughput than existing models.
What Problem Does This Paper Address?
In computational pathology, Whole Slide Images (WSIs) are "gigapixel" files—massive digital scans of tissue samples. Because these images are too large for standard neural networks, researchers use Multiple Instance Learning (MIL). In MIL, an image is broken into thousands of small patches (instances), which are then aggregated to make a single slide-level diagnosis.
Currently, the gold standard for aggregation is the Transformer. Transformers are excellent at modeling the correlations between different tissue patches, but they have a fatal flaw: quadratic complexity. As the number of patches increases, memory consumption and processing time grow exponentially ($O(N^2)$). For ultra-long sequences typical of large biopsies, Transformer-based MIL models often become too slow or memory-hungry for practical clinical deployment.
Methods
The authors propose RetMIL, which swaps out the standard Transformer backbone for a Retention Mechanism (inspired by the RetNet architecture used in Large Language Models).
The Core Innovation: Retention
Unlike self-attention, which compares every patch to every other patch simultaneously, the retention mechanism uses a linear approach. It incorporates a "relative distance decay matrix," which allows the model to weigh the importance of patches based on their sequence while maintaining a computational complexity that scales linearly ($O(N)$) with the number of patches.
Hierarchical Architecture
RetMIL processes the WSI sequence in two distinct stages:
- Local Level: The WSI sequence is divided into smaller subsequences (e.g., 512 patches). These are updated in parallel using a retention mechanism and then condensed into a single "local" feature embedding via attention pooling.
- Global Level: These local embeddings are fused into a global sequence. A serial retention mechanism then processes this global context to capture long-range dependencies across the entire slide.
- Classification: Finally, a global attention pooling layer produces the final slide-level representation for the classifier.
Datasets
The model was evaluated on three major benchmarks:
- CAMELYON (16 & 17): A dataset focused on detecting lymph node metastases in breast cancer (binary classification).
- BRACS: A multi-classification dataset for breast cancer subtyping.
- LUNG: A binary classification task for non-small cell lung cancer subtypes (LUAD vs. LUSC), using data from the TCGA repository for training and an internal "cooperative hospital" dataset for testing.
Key Insights and Ideas
- Breaking the Memory Wall: The most striking insight is that RetMIL maintains "almost constant" GPU memory usage regardless of the sequence length. While models like HIPT or HAG-MIL see memory costs skyrocket as slides get larger, RetMIL remains flat and efficient.
- The Best of Both Worlds: Retention acts as a hybrid between a Transformer and a Recurrent Neural Network (RNN). It offers the parallel training capabilities of Transformers but the efficient, $O(1)$ inference of RNNs.
- Spatial Awareness via Sequence: By splitting the WSI into subsequences and using a decay-based retention, the model naturally learns to prioritize local clusters of tissue while still "remembering" the broader context of the slide.
Key Results
- Superior Performance: On the CAMELYON dataset, RetMIL achieved an F1-score of 87.24%, outperforming the popular TransMIL model by over 3%.
- Speed and Efficiency: RetMIL demonstrated a 1.5x lead in throughput compared to TransMIL.
- Robustness to Length: In tests involving "ultra-long" sequences (over 15,000 patches), RetMIL's performance remained stable, whereas Transformer-based models often saw performance degradation or reached hardware limits.
- Interpretability: Heatmap visualizations (Fig. 4) confirmed that the model's attention scores align closely with the cancer regions manually outlined by pathologists.
Limitations and Future Work
While RetMIL represents a significant leap in efficiency, some areas remain for exploration:
- Subsequence Sensitivity: The paper uses a fixed subsequence length of 512. The impact of varying this "local window" size on different tissue types isn't fully explored.
- Feature Extraction: Like most MIL methods, RetMIL relies on a frozen, pre-trained feature extractor (ViT-S/16). Future work could investigate if end-to-end training with the retention mechanism provides further gains.
- Sequence Ordering: WSIs are 2D structures, but MIL treats them as 1D sequences. While the retention mechanism handles sequences well, exploring 2D-aware retention could be a promising next step.
Summary updated