Abstract
Histopathological whole slide image (WSI) analysis with deep learning has become a research focus in computational pathology. The current paradigm is mainly based on multiple instance learning (MIL), in which approaches with Transformer as the backbone are well discussed. These methods convert WSI tasks into sequence tasks by representing patches as tokens in the WSI sequence. However, the feature complexity brought by high heterogeneity and the ultra-long sequences brought by gigapixel size makes Transformer-based MIL suffer from the challenges of high memory consumption, slow inference speed, and lack of performance. To this end, we propose a retentive MIL method called RetMIL, which processes WSI sequences through hierarchical feature propagation structure. At the local level, the WSI sequence is divided into multiple subsequences. Tokens of each subsequence are updated through a parallel linear retention mechanism and aggregated utilizing an attention layer. At the global level, subsequences are fused into a global sequence, then updated through a serial retention mechanism, and finally the slide-level representation is obtained through a global attention pooling. We conduct experiments on two public CAMELYON and BRACS datasets and an public-internal LUNG dataset, confirming that RetMIL not only achieves state-of-the-art performance but also significantly reduces computational overhead.
Introduction
In computational pathology, Whole Slide Images (WSIs) are "gigapixel" files—massive digital scans of tissue samples. Because these images are too large for standard neural networks, researchers use Multiple Instance Learning (MIL). In MIL, an image is broken into thousands of small patches (instances), which are then aggregated to make a single slide-level diagnosis. Currently, the gold standard for aggregation is the Transformer. While Transformers excel at modeling correlations between tissue patches, they suffer from quadratic complexity ($O(N^2)$). As the number of patches increases, memory consumption and processing time grow exponentially, making Transformer-based MIL models difficult to deploy in clinical settings for ultra-long sequences.
Methodology: Hierarchical Retention
RetMIL addresses these limitations by replacing the standard Transformer backbone with a Retention Mechanism, inspired by the RetNet architecture. Unlike self-attention, which compares every patch to every other patch simultaneously, the retention mechanism uses a linear approach ($O(N)$). It incorporates a "relative distance decay matrix," allowing the model to weigh the importance of patches based on their sequence while maintaining computational efficiency.
The architecture processes the WSI sequence in two distinct stages:
- Local Level: The WSI sequence is divided into smaller subsequences. These are updated in parallel using a retention mechanism and condensed into a single "local" feature embedding via attention pooling.
- Global Level: These local embeddings are fused into a global sequence. A serial retention mechanism then processes this global context to capture long-range dependencies across the entire slide.
- Classification: A global attention pooling layer produces the final slide-level representation for a linear classifier.
Experimental Results and Efficiency
The model was evaluated on three major benchmarks: CAMELYON (16 & 17), BRACS, and a LUNG dataset (LUAD vs. LUSC). RetMIL achieved state-of-the-art performance, notably reaching an F1-score of 87.24% on CAMELYON, outperforming the popular TransMIL model by over 3%.
Beyond accuracy, the most striking advantage of RetMIL is its efficiency. It maintains nearly constant GPU memory usage regardless of the sequence length, whereas competing models like HIPT or HAG-MIL see memory costs skyrocket as slides get larger.
RetMIL also demonstrated a 1.5x lead in throughput compared to TransMIL. In tests involving ultra-long sequences (over 15,000 patches), RetMIL's performance remained stable, whereas Transformer-based models often reached hardware limits or suffered performance degradation.
Conclusion
RetMIL represents a significant leap in scaling histopathology analysis. By combining the parallel training capabilities of Transformers with the efficient inference of linear-complexity mechanisms, it breaks the "memory wall" associated with gigapixel WSI processing. Future work will explore the impact of varying subsequence lengths and the potential for end-to-end training with the retention mechanism to further enhance diagnostic performance.
Summary updated