Abstract
Automated medical report generation from X-ray images is a critical yet challenging task in clinical AI. Current models often struggle with shallow reasoning and repetitive text generation. We present ReMRG-XR, a novel framework that utilizes X-linear attention to capture high-order multi-modal interactions and a specialized repetition penalty within a reinforcement learning objective. Our approach significantly improves report quality, diversity, and clinical relevance, outperforming state-of-the-art baselines on the IU X-Ray and MIMIC-CXR datasets.
Introduction
Radiologists face an increasing burden of report writing, necessitating efficient AI-driven assistance. However, existing medical report generation models frequently encounter two primary obstacles: shallow reasoning and the "repetition trap." Standard attention mechanisms often fail to capture the nuanced, high-order relationships between visual features and medical terminology. Furthermore, optimization for standard metrics like CIDEr via Reinforcement Learning (RL) often encourages the model to repeat "safe" phrases, leading to redundant and less readable reports.
Methodology: The ReMRG-XR Framework
The ReMRG-XR model enhances the traditional encoder-decoder architecture through two key innovations: X-linear attention and a self-critical sequence training (SCST) strategy with a repetition penalty.
X-Linear Attention
Unlike standard "first-order" attention, which calculates simple correlations, X-linear attention employs bilinear pooling-based blocks. This allows the model to compute the outer product between feature vectors, capturing high-order interactions across different information channels. This sophisticated multi-modal reasoning is applied during both image encoding and report decoding, enabling the model to better resolve fine-grained anatomical details.
RL with Repetition Penalty
To address the issue of "stuttering" in generated reports, we incorporate a Repetition Penalty into the SCST process. During training, the model tracks generated trigrams. If a trigram is repeated, the model penalizes the log-probability of the constituent words. This forces the generator to explore more diverse and coherent linguistic structures while still optimizing for overall accuracy metrics.
Experimental Results
We evaluated ReMRG-XR on the IU X-Ray and MIMIC-CXR datasets.
Quantitative Performance
ReMRG-XR demonstrated superior performance across all standard NLP metrics. On the IU X-Ray dataset, it achieved a BLEU-4 score of 0.140 (vs. 0.079 for the Top-Down baseline). On the larger MIMIC-CXR dataset, it reached a BLEU-4 of 0.157 (vs. 0.093 for the baseline).
Qualitative Analysis
Qualitative comparisons highlight the model's ability to generate detailed and non-repetitive reports that closely align with human ground truth.
As shown in Figure 2, while baseline models like Top-Down often miss specific findings or repeat generic phrases (e.g., "no pleural effusion or pneumothorax"), ReMRG-XR captures subtle details such as "mild degenerative changes" and maintains a natural narrative flow.
Discussion and Key Insights
Our findings suggest that high-order reasoning is essential for medical imaging tasks, where fine-grained relationships between structures are paramount. Additionally, we identify a "CIDEr Paradox": while optimizing for CIDEr improves keyword matching, it can degrade text quality without explicit diversity constraints like our repetition penalty.
Conclusion and Future Work
ReMRG-XR represents a significant step toward clinically useful automated radiology reporting. Future work will focus on direct clinical validation by radiologists and extending the model's reasoning capabilities to handle multi-view longitudinal studies.
Summary updated