MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models

Zijie Fang, Yifeng Wang, Zhi Wang, Jian Zhang, Xiangyang Ji, Yongbing Zhang

Abstract

Recently, pathological diagnosis has achieved superior performance by combining deep learning models with the multiple instance learning (MIL) framework using whole slide images (WSIs). However, the giga-pixeled nature of WSIs poses a great challenge for efficient MIL. Existing studies either do not consider global dependencies among instances, or use approximations such as linear attentions to model the pair-to-pair instance interactions, which inevitably brings performance bottlenecks. To tackle this challenge, we propose a framework named MamMIL for WSI analysis by cooperating the selective structured state space model (i.e., Mamba) with MIL, enabling the modeling of global instance dependencies while maintaining linear complexity. Specifically, considering the irregularity of the tissue regions in WSIs, we represent each WSI as an undirected graph. To address the problem that Mamba can only process 1D sequences, we further propose a topology-aware scanning mechanism to serialize the WSI graphs while preserving the topological relationships among the instances. Finally, in order to further perceive the topological structures among the instances and incorporate short-range feature interactions, we propose an instance aggregation block based on graph neural networks. Experiments show that MamMIL can achieve advanced performance than the state-of-the-art frameworks.

Introduction

In digital pathology, Whole Slide Images (WSIs) are "giga-pixel" files containing billions of pixels, making them too large for direct GPU processing. The standard approach is Multiple Instance Learning (MIL), where an image is partitioned into thousands of small patches (instances). The primary challenge in MIL is effectively aggregating these instances to capture the global clinical context.

While Transformers have been used to model instance interactions, their quadratic computational complexity ($O(N^2)$) becomes a bottleneck for slides with tens of thousands of patches. State Space Models (SSMs), specifically Mamba, offer linear efficiency ($O(N)$) but are inherently designed for 1D sequences. This makes them difficult to apply to the irregular 2D spatial structures of human tissue.

The MamMIL Framework

To bridge the gap between irregular tissue structures and the linear efficiency of SSMs, we propose MamMIL. The framework operates in two main stages: feature extraction/graph construction and feature aggregation.

The MamMIL framework overview, illustrating the two-stage process and the internal structure of the Topology-Aware Mamba (TA-Mamba) block.

1. Graph-Based Representation

Unlike traditional MIL methods that treat patches as a simple list, MamMIL represents a WSI as an undirected graph. Each patch is a node, and edges are established between the 8 nearest neighbors (kNN) based on spatial coordinates. This preserves the physical layout and local neighborhood context of the tissue.

2. Topology-Aware Mamba (TA-Mamba)

To enable Mamba to process the 2D graph structure, we introduce a Topology-Aware Scanning mechanism. We first compute a Minimum Spanning Tree (MST) of the WSI graph to identify the most significant connections. This tree is then serialized into four distinct 1D sequences using different traversal strategies: Index-order, Pre-order, Post-order, and Level-order. These sequences ensure that spatially proximal patches remain close in the serialized input, allowing Mamba to capture topological relationships effectively.

3. GNN-based Instance Aggregation (GIA)

While TA-Mamba handles long-range global dependencies, we incorporate a GIA block (based on DeepGCN) to refine local, short-range interactions. This hybrid approach prevents the "over-smoothing" common in deep GNNs while benefiting from their local feature extraction capabilities. The framework also utilizes the updated Mamba2 architecture for improved hardware utilization and larger state dimensions.

Experimental Results

MamMIL was evaluated on three major benchmarks: Camelyon16 (breast cancer metastasis detection), BRACS (breast carcinoma subtyping), and TCGA-LUAD (survival analysis). We tested the framework using various backbones, including ResNet-50, VIM, and VMAMBA.

Comprehensive performance comparison of MamMIL against state-of-the-art MIL methods across different backbones and datasets.

As shown in the results, MamMIL consistently outperforms state-of-the-art models like TransMIL and MambaMIL. For instance, on the BRACS-7 dataset with a ResNet-50 backbone, MamMIL achieved an AUC of 77.46%, surpassing the previous best of 76.56%. In survival analysis (TCGA-LUAD), it reached a C-Index of 67.18*, demonstrating its robustness across different clinical tasks.

Conclusion

MamMIL successfully combines the linear efficiency of State Space Models with the spatial awareness of Graph Neural Networks. By using MST-based serialization, it captures the complex topological structures of WSIs more effectively than previous sequence-based or attention-based methods. Future work will explore more deterministic root node selection for MST traversals and potential end-to-end training optimizations.

Summary updated

Abstract

Recently, pathological diagnosis has achieved superior performance by combining deep learning models with the multiple instance learning (MIL) framework using whole slide images (WSIs). However, the giga-pixeled nature of WSIs poses a great challenge for efficient MIL. Existing studies either do not consider global dependencies among instances, or use approximations such as linear attentions to model the pair-to-pair instance interactions, which inevitably brings performance bottlenecks. To tackle this challenge, we propose a framework named MamMIL for WSI analysis by cooperating the selective structured state space model (i.e., Mamba) with MIL, enabling the modeling of global instance dependencies while maintaining linear complexity. Specifically, considering the irregularity of the tissue regions in WSIs, we represent each WSI as an undirected graph. To address the problem that Mamba can only process 1D sequences, we further propose a topology-aware scanning mechanism to serialize the WSI graphs while preserving the topological relationships among the instances. Finally, in order to further perceive the topological structures among the instances and incorporate short-range feature interactions, we propose an instance aggregation block based on graph neural networks. Experiments show that MamMIL can achieve advanced performance than the state-of-the-art frameworks. The code can be accessed at https://github.com/Vison307/MamMIL.
BibTeX
@inproceedings{fang2024mammil,
  author = {Zijie Fang and Yifeng Wang and Zhi Wang and Jian Zhang and Xiangyang Ji and Yongbing Zhang},
  title = {{MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822552},
}
Figure