RRG-Mamba: Efficient Radiology Report Generation with State Space Model

Xiaodi Hou, Xiaobo Li, Mingyu Lu, Simiao Wang, Yijia ZhangIJCAI 2025 · DOI

TL;DR

This paper introduces RRG-Mamba, a novel framework for generating radiology reports from medical images. It addresses the trade-off between capturing long-range dependencies in images and maintaining computational efficiency by leveraging the Mamba state space model, enhanced with rotary position encoding and a global dependency learning module. RRG-Mamba achieves state-of-the-art performance on public datasets while significantly improving computational and memory efficiency.

What Problem Does This Paper Address?

Automatic radiology report generation (RRG) is a critical task in medical imaging, aiming to automate the creation of descriptive and clinically relevant reports from medical images. This process is crucial for improving diagnostic accuracy, enhancing efficiency in healthcare delivery, and alleviating the workload of radiologists. Traditional approaches often rely on CNNs for image feature extraction and Transformers for report generation. While these methods have shown promise, they struggle with two key challenges:

  1. Balancing Global Dependencies and Computational Efficiency: Capturing long-range spatial dependencies within medical images is essential for accurate diagnosis. However, models like Vision Transformers (ViTs) that excel at this often incur high computational costs (quadratic complexity) due to their self-attention mechanisms, making them impractical for large-scale datasets.
  2. Modeling Complex Spatial Relationships: Standard CNNs are good at local feature extraction but have limited receptive fields, hindering their ability to model global context. While Transformers can capture global dependencies, their computational demands remain a bottleneck.

This paper aims to bridge this gap by proposing a framework that efficiently models long-range dependencies in visual features while maintaining computational tractability.

Methods

RRG-Mamba is built upon the Mamba state space model, known for its linear complexity in modeling long sequences. The authors introduce several key enhancements:

  • Mamba Backbone: Mamba is chosen for its efficiency in handling long sequences and its potential for capturing global dependencies.
  • Rotary Position Encoding (RoPE): To address Mamba's limitations in capturing complex spatial dependencies due to its fixed positional encoding, RoPE is integrated. RoPE allows for dynamic modeling of relative positional information within visual feature sequences, enhancing the model's ability to understand spatial relationships.
  • Global Dependency Learning Module (GDLM): This module is designed to dynamically capture relative positional information and effectively model complex spatial relationships. It combines RoPE with a 2D selective scan (SS2D) layer, inspired by Mamba's architecture, to process visual features from multiple directions and capture long-distance dependencies.
  • Dynamic Position Bias (DPB): The GDLM incorporates DPB to adjust the spatial information of serialized tokens, making the model more sensitive to specific spatial regions within the visual features.
  • Cross-Modal Adapter (CMA): To improve the fusion of visual and textual information, a CMA is introduced. This module leverages Chexpert to generate pseudo-labels for medical images, aiding in semantic alignment between modalities and enhancing cross-modal consistency.
  • Encoder-Decoder Architecture: A standard Transformer-based encoder-decoder architecture is used for report generation, where the encoder processes visual features and the decoder generates the report token by token.

Datasets

The proposed RRG-Mamba model is evaluated on two widely used, publicly available radiology report generation datasets:

  • IU X-Ray: A dataset from Indiana University.
  • MIMIC-CXR: A large-scale chest X-ray dataset.

The paper does not introduce new benchmarks but focuses on evaluating its model against existing state-of-the-art methods on these established datasets.

Key Insights and Ideas

  • Mamba's Potential for RRG: The paper demonstrates that Mamba, a recent advancement in sequence modeling, can be effectively adapted for the complex task of radiology report generation, offering a promising alternative to computationally expensive Transformers.
  • RoPE is Crucial for Spatial Understanding: Integrating RoPE into Mamba is shown to be vital for improving its ability to capture fine-grained spatial relationships in visual features, which is a known limitation of vanilla Mamba for image-related tasks.
  • GDLM as a Powerful Feature Extractor: The GDLM, with its combination of RoPE and SS2D, effectively models long-range dependencies and complex spatial structures, leading to richer visual representations.
  • Efficiency Gains are Significant: The RRG-Mamba framework achieves substantial improvements in computational and memory efficiency compared to Transformer-based models, making it more practical for real-world applications.
  • Clinical Utility Matters: Beyond standard NLG metrics, the paper emphasizes the importance of clinical efficacy, showing that RRG-Mamba generates reports with higher accuracy and relevance.

Key Results

  • Performance Improvement: RRG-Mamba achieves a 3.7% improvement in BLEU-4 score over existing models on the IU X-Ray dataset. It also shows significant gains on the MIMIC-CXR dataset.
  • Efficiency: The model demonstrates superior computational and memory efficiency, with faster training times and reduced parameter counts compared to Transformer-based approaches. For instance, the DCL model with a ViT encoder has 66.56MB parameters and takes 1.33 minutes per epoch, while RRG-Mamba with DenseNet-121 has 134.33MB parameters but achieves better performance with 2.29 minutes per epoch, indicating a better trade-off.
  • Ablation Studies: Removing key components like GDLM or RoPE leads to a significant drop in performance, highlighting their critical contributions.
  • Qualitative Results: Visualizations show that RRG-Mamba generates more accurate and detailed semantic descriptions, better capturing subtle lesions and distinguishing them from normal findings compared to Transformer and R2GenCMN.

Limitations and Future Work

  • Hyperparameter Sensitivity: The performance of RRG-Mamba is sensitive to the hyperparameter k (word identification ratio), with an optimal value of 0.5. Values too high or too low can degrade performance by introducing redundancy or omitting critical terms.
  • Visual Encoder Choice: While RRG-Mamba performs well with various visual encoders, the paper notes that CNN-based encoders (like ResNet-101 and DenseNet-121) offer superior reasoning capabilities and training efficiency compared to ViT-based encoders for this task.
  • Future Directions: The authors suggest exploring different versions of GDLM with varying model capacities to further optimize performance. They also highlight the potential of combining Mamba with other advanced techniques for even better report generation. The paper's success with Mamba opens avenues for further research into state space models for medical imaging tasks.

Summary updated

BibTeX
@inproceedings{hou2025rrg,
  author = {Xiaodi Hou and Xiaobo Li and Mingyu Lu and Simiao Wang and Yijia Zhang},
  title = {{RRG-Mamba: Efficient Radiology Report Generation with State Space Model}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/824},
}
Figure