Reinforced Medical Report Generation with X-Linear Attention and Repetition Penalty

Wenting Xu, Chang Qi, Zhenghua Xu, Thomas LukasiewiczarXiv 2020 · OpenReview

Abstract

Automated medical report generation from X-ray images is a critical yet challenging task in clinical AI. Current models often struggle with shallow reasoning and repetitive text generation. We present ReMRG-XR, a novel framework that utilizes X-linear attention to capture high-order multi-modal interactions and a specialized repetition penalty within a reinforcement learning objective. Our approach significantly improves report quality, diversity, and clinical relevance, outperforming state-of-the-art baselines on the IU X-Ray and MIMIC-CXR datasets.

Introduction

Radiologists face an increasing burden of report writing, necessitating efficient AI-driven assistance. However, existing medical report generation models frequently encounter two primary obstacles: shallow reasoning and the "repetition trap." Standard attention mechanisms often fail to capture the nuanced, high-order relationships between visual features and medical terminology. Furthermore, optimization for standard metrics like CIDEr via Reinforcement Learning (RL) often encourages the model to repeat "safe" phrases, leading to redundant and less readable reports.

Methodology: The ReMRG-XR Framework

The ReMRG-XR model enhances the traditional encoder-decoder architecture through two key innovations: X-linear attention and a self-critical sequence training (SCST) strategy with a repetition penalty.

The ReMRG-XR architecture, featuring X-linear attention blocks in the encoder/decoder and a repetition penalty module in the RL loop.

X-Linear Attention

Unlike standard "first-order" attention, which calculates simple correlations, X-linear attention employs bilinear pooling-based blocks. This allows the model to compute the outer product between feature vectors, capturing high-order interactions across different information channels. This sophisticated multi-modal reasoning is applied during both image encoding and report decoding, enabling the model to better resolve fine-grained anatomical details.

RL with Repetition Penalty

To address the issue of "stuttering" in generated reports, we incorporate a Repetition Penalty into the SCST process. During training, the model tracks generated trigrams. If a trigram is repeated, the model penalizes the log-probability of the constituent words. This forces the generator to explore more diverse and coherent linguistic structures while still optimizing for overall accuracy metrics.

Experimental Results

We evaluated ReMRG-XR on the IU X-Ray and MIMIC-CXR datasets.

Quantitative Performance

ReMRG-XR demonstrated superior performance across all standard NLP metrics. On the IU X-Ray dataset, it achieved a BLEU-4 score of 0.140 (vs. 0.079 for the Top-Down baseline). On the larger MIMIC-CXR dataset, it reached a BLEU-4 of 0.157 (vs. 0.093 for the baseline).

Qualitative Analysis

Qualitative comparisons highlight the model's ability to generate detailed and non-repetitive reports that closely align with human ground truth.

Qualitative comparison of reports generated by various models. ReMRG-XR produces more accurate and diverse descriptions compared to baselines.

As shown in Figure 2, while baseline models like Top-Down often miss specific findings or repeat generic phrases (e.g., "no pleural effusion or pneumothorax"), ReMRG-XR captures subtle details such as "mild degenerative changes" and maintains a natural narrative flow.

Discussion and Key Insights

Our findings suggest that high-order reasoning is essential for medical imaging tasks, where fine-grained relationships between structures are paramount. Additionally, we identify a "CIDEr Paradox": while optimizing for CIDEr improves keyword matching, it can degrade text quality without explicit diversity constraints like our repetition penalty.

Conclusion and Future Work

ReMRG-XR represents a significant step toward clinically useful automated radiology reporting. Future work will focus on direct clinical validation by radiologists and extending the model's reasoning capabilities to handle multi-view longitudinal studies.

Summary updated

Abstract

To reduce doctors' workload, deep-learning-based automatic medical report generation has recently attracted more and more research efforts, where attention mechanisms and reinforcement learning are integrated with the classic encoder-decoder architecture to enhance the performance of deep models. However, these state-of-the-art solutions mainly suffer from two shortcomings: (i) their attention mechanisms cannot utilize high-order feature interactions, and (ii) due to the use of TF-IDF-based reward functions, these methods are fragile with generating repeated terms. Therefore, in this work, we propose a reinforced medical report generation solution with x-linear attention and repetition penalty mechanisms (ReMRG-XR) to overcome these problems. Specifically, x-linear attention modules are used to explore high-order feature interactions and achieve multi-modal reasoning, while repetition penalty is used to apply penalties to repeated terms during the model's training process. Extensive experimental studies have been conducted on two public datasets, and the results show that ReMRG-XR greatly outperforms the state-of-the-art baselines in terms of all metrics.
BibTeX
@misc{xu2020reinforced,
  author = {Wenting Xu and Chang Qi and Zhenghua Xu and Thomas Lukasiewicz},
  title = {{Reinforced Medical Report Generation with X-Linear Attention and Repetition Penalty}},
  year = {2020},
  eprint = {2011.07680},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.07680},
}
Figure