Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading

Qi Bi, Jingjun Yi, Hao Zheng, Wei Ji, Haolan Zhan, Yawen Huang, Yuexiang Li, Yefeng ZhengNeurIPS 2024 · OpenReview

Assessing the severity of a disease from a medical image—known as disease grading—is a high-stakes task that is notoriously difficult for AI. Unlike simple classification (e.g., "cat vs. dog"), disease grading involves a continuous spectrum where the lines between "moderate" and "severe" are often blurred.

A new paper, titled Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading, introduces a novel approach to tackle two of the biggest hurdles in the field: identifying tiny but critical lesions and ensuring models work across different hospitals and scanners (cross-domain generalization).

TL;DR

Samba combines the efficiency of State Space Models (Mamba) with an Expectation-Maximization (EM) recalibration mechanism to improve medical image grading. It significantly outperforms existing baselines by focusing on small, decisive pathological regions and mapping features into a domain-invariant, compact space.

What Problem Does This Paper Address?

Disease grading faces two primary challenges:

  1. Inherent Ambiguity: Disease progression is a continuous process. As illustrated below, there is high "within-level discrepancy" (two images of the same grade looking remarkably different) and "cross-level similarity" (adjacent grades looking nearly identical).
  2. Domain Shift: A model trained on high-quality images from one clinic often fails when deployed at another clinic using different scanners or lighting. Standard models tend to cluster images by their source domain rather than their disease severity, leading to poor generalization.
Challenges in disease grading

The authors argue that the key to solving this is accurately identifying the most severe lesions, which often occupy only a tiny fraction of the image (like a small hemorrhage in a retinal scan).

Methods

The Samba architecture introduces two novel components to the standard vision pipeline:

1. Recurrent Patch Modeling (via Mamba)

Instead of treating an image as a bag of patches (like a standard Vision Transformer), Samba treats image patches as a sequence. It utilizes a bidirectional Mamba (a type of State Space Model or SSM).

  • Why? Recurrent modeling allows the network to store information about critical, tiny lesions in "hidden states" and propagate that information across the entire sequence.
  • The Benefit: This ensures that even if a decisive lesion appears in only a few pixels, its "severity signal" influences the final global representation.

2. EM-based State Recalibration

To handle domain shift, the authors designed a module that uses the Expectation-Maximization (EM) algorithm.

  • They model the distribution of lesions using a Gaussian Mixture Model (GMM) with learnable "severity bases."
  • The EM module iteratively maps patch embeddings into a more compact, domain-invariant space. This effectively "filters out" the stylistic noise of a specific scanner and focuses on the underlying pathological features.

Datasets

The authors evaluated Samba on three distinct cross-domain benchmarks:

  • Fatigue Fracture Grading: X-ray images from two different hospitals.
  • Breast Cancer Grading: H&E stained pathological images at different magnifications (20x vs. 40x).
  • Diabetic Retinopathy (DR) Grading: A massive benchmark involving six different datasets (APTOS, DeepDR, Messidor, IDRID, FGADR, and RLDR) to test single-source domain generalization.

Key Insights and Ideas

  • Recurrence over Attention: While Transformers are popular, their global attention can sometimes "wash out" tiny, high-importance details. Samba’s recurrent hidden states act as a memory buffer specifically tuned to catch these "decisive lesions."
  • Clustering by Severity, Not Style: One of the most striking findings is how the model organizes data. Standard models often cluster data by which hospital it came from. Samba successfully maps the feature space so that data clusters by disease grade, regardless of the source domain.
Feature Space Illustration
  • Efficiency: By using an SSM instead of standard Transformers, the model maintains a better balance between computational efficiency and the ability to process long sequences of image patches.

Key Results

Samba demonstrated substantial performance gains over the state-of-the-art VMamba baseline:

  • Fatigue Fracture: Improved average accuracy by 23.5%.
  • Breast Cancer: Improved average accuracy by 5.6%.
  • Diabetic Retinopathy: Improved average accuracy by 4.1%.

In the DR benchmark, Samba achieved the "best" or "second best" results across almost every unseen dataset, proving its robustness to domain shift and its ability to cleanly separate disease levels.

Limitations and Future Work

The authors note a few areas for improvement:

  • Class Imbalance: If the source training data is heavily imbalanced (e.g., mostly healthy cases), the GMM might struggle to accurately estimate the distribution of rare, severe lesions.
  • 3D Extension: Currently, Samba is designed for 2D images. Extending this recurrent modeling to 3D volumes (like CT or MRI scans) is a promising but computationally challenging next step.
  • Complexity: The EM-based recalibration adds an iterative step during the forward pass, which, while effective, increases the complexity of the training loop.

Overall, Samba represents a significant step forward in making medical AI more reliable and generalizable for real-world clinical use.

Summary updated

Abstract

Disease grading is a crucial task in medical image analysis. Due to the continuous progression of diseases, i.e., the variability within the same level and the similarity between adjacent stages, accurate grading is highly challenging. Furthermore, in real-world scenarios, models trained on limited source domain datasets should also be capable of handling data from unseen target domains. Due to the cross-domain variants, the feature distribution between source and unseen target domains can be dramatically different, leading to a substantial decrease in model performance. To address these challenges in cross-domain disease grading, we propose a Severity-aware Recurrent Modeling (Samba) method in this paper. As the core objective of most staging tasks is to identify the most severe lesions, which may only occupy a small portion of the image, we propose to encode image patches in a sequential and recurrent manner. Specifically, a state space model is tailored to store and transport the severity information by hidden states. Moreover, to mitigate the impact of cross-domain variants, an Expectation-Maximization (EM) based state recalibration mechanism is designed to map the patch embeddings into a more compact space. We model the feature distributions of different lesions through the Gaussian Mixture Model (GMM) and reconstruct the intermediate features based on learnable severity bases. Extensive experiments show the proposed Samba outperforms the VMamba baseline by an average accuracy of 23.5\%, 5.6\% and 4.1\% on the cross-domain grading of fatigue fracture, breast cancer and diabetic retinopathy, respectively. Source code is available at \url{https://github.com/BiQiWHU/Samba}.
BibTeX
@inproceedings{bi2024samba,
  author = {Qi Bi and Jingjun Yi and Hao Zheng and Wei Ji and Haolan Zhan and Yawen Huang and Yuexiang Li and Yefeng Zheng},
  title = {{Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=aIeXn5103e},
}
Figure