MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models

Zijie Fang, Yifeng Wang, Ye Zhang, Zhi Wang, Jian Zhang, Xiangyang Ji, Yongbing Zhang

TL;DR

MamMIL is a new framework for analyzing massive Whole Slide Images (WSIs) by combining the linear efficiency of Mamba2 (State Space Models) with the spatial awareness of Graph Neural Networks. By converting irregular tissue structures into topology-aware sequences via Minimum Spanning Trees, it captures both local and global clinical context more effectively than previous Transformer-based methods.

What Problem Does This Paper Address?

In digital pathology, Whole Slide Images (WSIs) are "giga-pixel" files—single images containing billions of pixels. Because they are too large to fit into GPU memory, researchers use Multiple Instance Learning (MIL): they break the image into thousands of small patches (instances) and try to aggregate them to make a diagnosis.

The challenge lies in context. Early MIL models treated patches as independent, ignoring how cells interact. Later, Transformers were used to model these interactions, but their "self-attention" mechanism is computationally expensive (quadratic complexity), making it slow for slides with tens of thousands of patches. While newer "State Space Models" (like Mamba) offer linear efficiency, they are designed for 1D sequences (like text), making them a poor fit for the irregular, 2D spatial structures found in human tissue.

Methods

The authors introduce MamMIL, which treats a WSI as an undirected graph rather than a simple list of patches. The architecture consists of three main innovations:

  1. Graph Representation: Each patch is a node, and edges are created between the 8 nearest neighbors (kNN) based on spatial coordinates. This preserves the physical layout of the tissue.
  2. Topology-Aware Mamba (TA-Mamba): To let Mamba "read" a 2D graph, the authors use a Minimum Spanning Tree (MST) to find the most efficient connections. They then serialize this tree into four different 1D sequences using different traversal strategies (Pre-order, Post-order, Level-order, and Index-order). This ensures that patches that are physically close in the tissue stay close in the sequence.
  3. GNN-based Instance Aggregation (GIA): While Mamba handles global, long-range dependencies, the authors insert a GIA block (based on DeepGCN) to specifically refine local, short-range interactions between neighboring patches.
  4. Mamba2 Integration: The framework utilizes the updated Mamba2 architecture, which offers larger state dimensions and better hardware utilization than the original Mamba.

Datasets

The model was evaluated on three major public benchmarks:

  • Camelyon16: A dataset for detecting breast cancer metastasis in lymph nodes (Classification).
  • BRACS: A large-scale dataset for breast carcinoma subtyping (Classification).
  • TCGA-LUAD: Data from The Cancer Genome Atlas used for predicting patient survival times (Survival Analysis).

The authors tested MamMIL using three different feature extractors: standard ResNet-50 and two Mamba-based vision backbones (VIM and VMAMBA).

Key Insights and Ideas

  • MST as a Bridge: The "aha moment" is using the Minimum Spanning Tree to flatten a graph. It provides a mathematically grounded way to turn a complex 2D structure into a 1D string that a sequence model can understand without losing the "neighborhood" context.
  • Hybrid Local-Global Modeling: The paper demonstrates that you don't have to choose between GNNs and SSMs. GNNs are great at local texture, but they suffer from "over-smoothing" if they get too deep. By using GNNs for local blocks and Mamba for global blocks, MamMIL gets the best of both worlds.
  • Multi-Traversal Robustness: By scanning the MST in four different ways (e.g., top-down vs. leaf-to-root), the model views the tissue structure from multiple "perspectives," leading to more robust feature fusion.

Key Results

MamMIL consistently outperformed State-of-the-Art (SOTA) models like TransMIL and MambaMIL:

  • Classification: On the BRACS-7 dataset using a ResNet-50 backbone, MamMIL achieved an AUC of 77.46%*, a significant jump over the previous SOTA of 76.56%.
  • Survival Analysis: On TCGA-LUAD, it reached a C-Index of 67.18, outperforming other models by a margin of ~2-3 points.
  • Ablation: Removing the GIA block or the topology-aware scanning significantly dropped performance, proving that the spatial graph structure is critical for accuracy.

Limitations and Future Work

While MamMIL is highly effective, the authors note a few areas for improvement:

  • Root Node Selection: Currently, the root node for the MST traversal is selected randomly. A more deterministic or "clinically informed" selection (e.g., starting from a high-attention region) might improve stability.
  • Computational Overhead: While Mamba is linear, constructing kNN graphs and MSTs for every slide adds a pre-processing step that could be optimized.
  • Feature Extraction: The model still relies on frozen, pre-trained feature extractors. End-to-end training of the entire pipeline remains a "holy grail" in the field due to memory constraints.

Summary updated

Abstract

Recently, pathological diagnosis has achieved superior performance by combining deep learning models with the multiple instance learning (MIL) framework using whole slide images (WSIs). However, the giga-pixeled nature of WSIs poses a great challenge for efficient MIL. Existing studies either do not consider global dependencies among instances, or use approximations such as linear attentions to model the pair-to-pair instance interactions, which inevitably brings performance bottlenecks. To tackle this challenge, we propose a framework named MamMIL for WSI analysis by cooperating the selective structured state space model (i.e., Mamba) with MIL, enabling the modeling of global instance dependencies while maintaining linear complexity. Specifically, considering the irregularity of the tissue regions in WSIs, we represent each WSI as an undirected graph. To address the problem that Mamba can only process 1D sequences, we further propose a topology-aware scanning mechanism to serialize the WSI graphs while preserving the topological relationships among the instances. Finally, in order to further perceive the topological structures among the instances and incorporate short-range feature interactions, we propose an instance aggregation block based on graph neural networks. Experiments show that MamMIL can achieve advanced performance than the state-of-the-art frameworks. The code can be accessed at https://github.com/Vison307/MamMIL.
BibTeX
@inproceedings{fang2024mammil,
  author = {Zijie Fang and Yifeng Wang and Ye Zhang and Zhi Wang and Jian Zhang and Xiangyang Ji and Yongbing Zhang},
  title = {{MamMIL: Multiple Instance Learning for Whole Slide Images with State Space Models}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822552},
}
Figure