Abstract
Inspired by the tremendous success of Large Language Models (LLMs), existing X-ray medical report generation methods attempt to leverage large models to achieve better performance. They usually adopt a Transformer to extract the visual features of a given X-ray image, and then, feed them into the LLM for text generation. How to extract more effective information for the LLMs to help them improve final results is an urgent problem that needs to be solved. Additionally, the use of visual Transformer models also brings high computational complexity. To address these issues, this paper proposes a novel context-guided efficient X-ray medical report generation framework. Specifically, we introduce the Mamba as the vision backbone with linear complexity, and the performance obtained is comparable to that of the strong Transformer model. More importantly, we perform context retrieval from the training set for samples within each mini-batch during the training phase, utilizing both positively and negatively related samples to enhance feature representation and discriminative learning. Subsequently, we feed the vision tokens, context information, and prompt statements to invoke the LLM for generating high-quality medical reports. Extensive experiments on three X-ray report generation datasets (i.e., IU-Xray, MIMIC-CXR, CheXpert Plus) fully validated the effectiveness of our proposed model.
Introduction
Automating the generation of radiology reports from X-ray images is a high-stakes task that can significantly alleviate clinical workloads. While Large Language Models (LLMs) have shown promise in this area, two major bottlenecks remain: computational complexity and contextual blindness. Most existing methods use Vision Transformers (ViTs) to extract image features, but the self-attention mechanism has quadratic complexity ($O(N^2)$), making it slow for high-resolution medical images. Furthermore, LLMs often struggle to distinguish subtle pathological changes from normal anatomical structures when given only raw visual tokens, lacking a "baseline" of what a healthy or diseased version of that specific patient's anatomy might look like.
Methodology: Contextual and Residual Information Mining
The proposed R2GenCSR framework addresses these issues by combining an efficient vision backbone with a novel context-retrieval strategy. The architecture consists of three main components:
Mamba Vision Backbone
Instead of a standard Transformer, R2GenCSR uses VMamba, a State Space Model (SSM) with linear complexity ($O(N)$). This allows for efficient processing of long-range visual dependencies in high-resolution X-rays, reducing training time per epoch from 5.85 hours (with Swin Transformer) to 3.98 hours on the same hardware.
Contextual Sample Retrieval and Residual Calculation
During training and inference, the system retrieves "context samples" from the training set, specifically looking for positive samples (images with confirmed diseases) and negative samples (normal images). The core innovation is the calculation of residual tokens: the model subtracts the features of the retrieved "normal" and "diseased" samples from the current image's features in the embedding space. This "delta" highlights exactly what is unique or abnormal about the current scan, providing the LLM with a clear comparative baseline.
Experimental Results
R2GenCSR was evaluated on three major benchmarks: IU X-Ray, MIMIC-CXR, and CheXpert Plus. The model achieved state-of-the-art (SOTA) results across all datasets, demonstrating both linguistic fluency and clinical accuracy.
On the IU X-Ray dataset, R2GenCSR achieved a BLEU-4 score of 0.206 and a CIDEr score of 0.579, outperforming previous LLM-based methods like R2GenGPT. On the large-scale MIMIC-CXR dataset, it reached a BLEU-4 of 0.136. Beyond standard metrics, the model significantly improved the F1-score for disease detection (0.484) on MIMIC-CXR, suggesting that the generated reports are highly reliable for clinical decision support.
Conclusion
R2GenCSR demonstrates that the power of comparison is crucial for medical AI. By providing LLMs with "residual" information—the difference between a patient's scan and healthy/diseased references—the model can focus its attention on pathological deviations. Combined with the efficiency of the Mamba backbone, R2GenCSR sets a new standard for automated, high-fidelity radiology report generation.
Summary updated