Hybrid Reinforced Medical Report Generation With M-Linear Attention and Repetition Penalty

Zhenghua Xu, Wenting Xu, Ruizhi Wang, Junyang Chen, Chang Qi, Thomas Lukasiewicz

TL;DR

This paper introduces HReMRG-MR, a novel method for automatically generating medical reports from X-ray images. It tackles key limitations of existing methods by employing a hybrid reward system for more comprehensive optimization, a novel "m-linear attention" mechanism for better feature understanding, and a repetition penalty to improve report readability, achieving state-of-the-art results on public datasets.

What Problem Does This Paper Address?

The increasing volume of medical imaging examinations places a significant burden on radiologists, leading to potential burnout and an increased risk of misdiagnosis. Automating the generation of medical reports, which describe findings from these images, is a critical need. While deep learning models, particularly those combining Convolutional Neural Networks (CNNs) for image encoding and Recurrent Neural Networks (RNNs) for report decoding, have shown promise, they suffer from several shortcomings:

  • Incomprehensive Optimization: Existing methods often rely on a single or a few metrics (like CIDEr) for optimization, which doesn't fully capture the nuances of long-form text generation and can lead to suboptimal performance across various aspects of report quality.
  • Low-Order Attention Mechanisms: Current attention mechanisms often focus on simple spatial or semantic relationships, failing to capture higher-order feature interactions within images and between images and text, which are crucial for identifying subtle abnormalities.
  • Repetitive Generation: A common issue is the generation of repetitive phrases, especially in "normal finding" sections, which degrades the coherence and readability of the reports.

Methods

The proposed Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty (HReMRG-MR) addresses these issues through three core components:

  1. Hybrid Weighted Reward: Instead of relying on a single metric, HReMRG-MR uses a combination of multiple natural language generation metrics as rewards. To find the optimal balance, they introduce a search algorithm with linear complexity to efficiently determine the best weight combination for these metrics, aiming for a more balanced and comprehensive optimization.
  2. M-Linear Attention Mechanism: This novel attention module enhances the model's ability to understand complex relationships. It builds upon bi-linear pooling, which captures second-order feature interactions, and stacks these blocks to explore high-order intra-modal (within image features) and inter-modal (between image and text features) interactions. This allows for a more nuanced understanding of visual information and its correlation with textual descriptions.
  3. Repetition Penalty Module: To combat repetitive phrasing, an exponential weighted penalty is applied during the generation process. This mechanism penalizes repeated trigrams, encouraging the model to produce more diverse and coherent sentences, thereby improving the overall readability of the generated reports.

The architecture itself is a CNN-RNN encoder-decoder. A ResNet-101 is used as the visual encoder, followed by the proposed m-linear attention blocks. The decoder is an LSTM that also incorporates m-linear attention to fuse visual and textual information. Reinforcement learning, specifically the Self-Critical Sequence Training (SCST) algorithm, is employed to optimize the model using the hybrid reward.

Datasets

The paper evaluates HReMRG-MR on two publicly available chest X-ray image datasets:

  • IU X-Ray: Contains 7,470 medical images and 3,955 reports. After filtering, 6,222 images and 3,111 paired reports were used, focusing on the "Impression" and "Findings" sections.
  • MIMIC-CXR: A much larger dataset with 377,110 images and 227,827 reports. The study used 76,724 paired images and reports, also focusing on "Impression" and "Findings."

For both datasets, standard preprocessing steps like lowercasing, tokenization, and filtering infrequent words were applied. The data was split into training, validation, and testing sets with a 7:1:2 ratio.

Key Insights and Ideas

  • Hybrid Rewards are Crucial for Medical Reports: The paper strongly suggests that relying on a single metric like CIDEr is insufficient for optimizing medical report generation. A hybrid reward system, carefully weighted, leads to more balanced and superior performance across various quality aspects.
  • High-Order Feature Interactions Matter: The m-linear attention mechanism's success highlights the importance of capturing complex, high-order relationships within image features and between image and text. This goes beyond simple attention and is key to identifying subtle abnormalities.
  • Repetition is a Real Problem, and Penalty Works: The repetition penalty module is shown to be effective in generating more diverse and readable reports, directly addressing a significant flaw in many existing models.
  • Efficient Weight Search is Necessary: Finding the optimal weights for a hybrid reward can be computationally expensive. The proposed linear-complexity search algorithm is a practical contribution for making this process feasible.

Key Results

HReMRG-MR significantly outperforms state-of-the-art baselines on both the IU X-Ray and MIMIC-CXR datasets across multiple evaluation metrics, including BLEU, CIDEr, METEOR, and ROUGE-L. For instance, on IU X-Ray, HReMRG-MR achieved a CIDEr score of 0.5239, compared to the best baseline (X-LAN) at 0.4508. Qualitative examples also show that HReMRG-MR generates more coherent and accurate reports. Ablation studies confirm the effectiveness of each proposed component: the hybrid reward, m-linear attention, and repetition penalty all contribute to the overall performance improvement.

Limitations and Future Work

The authors acknowledge that while their search solution for hybrid rewards is efficient, it is still time-consuming. They also note that there are no universally accepted metrics specifically designed for medical report generation, which makes comprehensive evaluation challenging. Future work could explore more effective and specialized metrics for this domain. Additionally, they suggest investigating inverse reinforcement learning to find more suitable rewards and further improve generation performance.

Summary updated

Abstract

To reduce doctors’ workload, deep-learning-based automatic medical report generation has recently attracted more and more research efforts, where deep convolutional neural networks (CNNs) are employed to encode the input images, and recurrent neural networks (RNNs) are used to decode the visual features into medical reports automatically. However, these state-of-the-art methods mainly suffer from three shortcomings: 1) incomprehensive optimization; 2) low-order and unidimensional attention; and 3) repeated generation. In this article, we propose a hybrid reinforced medical report generation method with m-linear attention and repetition penalty mechanism (HReMRG-MR) to overcome these problems. Specifically, a hybrid reward with different weights is employed to remedy the limitations of single-metric-based rewards, and a local optimal weight search algorithm is proposed to significantly reduce the complexity of searching the weights of the rewards from exponential to linear. Furthermore, we use m-linear attention modules to learn multidimensional high-order feature interactions and to achieve multimodal reasoning, while a new repetition penalty is proposed to apply penalties to repeated terms adaptively during the model’s training process. Extensive experimental studies on two public benchmark datasets show that HReMRG-MR greatly outperforms the state-of-the-art baselines in terms of all metrics. The effectiveness and necessity of all components in HReMRG-MR are also proved by ablation studies. Additional experiments are further conducted and the results demonstrate that our proposed local optimal weight search algorithm can significantly reduce the search time while maintaining superior medical report generation performances.
BibTeX
@article{xu2022hybrid,
  author = {Zhenghua Xu and Wenting Xu and Ruizhi Wang and Junyang Chen and Chang Qi and Thomas Lukasiewicz},
  title = {{Hybrid Reinforced Medical Report Generation With M-Linear Attention and Repetition Penalty}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=NnbaiwmYr4},
}
Figure