Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment

Jianbo Yuan, Haofu Liao, Rui Luo, Jiebo Luo

Abstract

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

Introduction

Writing radiology reports is a critical bottleneck in modern healthcare. It is a time-intensive process requiring high-level expertise to translate complex visual cues into precise clinical language. While AI has excelled in general image captioning, medical reporting presents unique challenges: the visual cues are subtle and overlapping, high-quality paired datasets are scarce, and the requirement for clinical precision is absolute. A minor error in a general caption is trivial, but misidentifying a pathology in a radiology report can lead to incorrect patient treatment.

Existing models often treat different X-ray views (e.g., frontal and lateral) as independent or struggle to maintain clinical accuracy over multi-sentence paragraphs. This paper introduces a framework that addresses these issues through multi-view image fusion and medical concept enrichment.

Methodology

The proposed architecture utilizes a generative encoder-decoder framework with several domain-specific enhancements designed to bridge the gap between visual features and clinical text.

System architecture for automatic radiology report generation.

Multi-View Encoder and Fusion

The model employs a ResNet-152 backbone to process both frontal and lateral X-rays. To ensure robust feature extraction, the authors implement a Cross-View Consistency (CVC) loss, which forces the encoder to reach similar diagnostic conclusions regardless of the viewing angle. For text generation, the model uses a Late Fusion Attention mechanism. Instead of simply averaging features, a sentence-level attention model selectively "looks" at the most relevant parts of both views for each sentence it generates.

Hierarchical Decoder and Concept Enrichment

To manage long-form text, a two-tiered LSTM decoder is used: a Sentence LSTM determines the topic, and a Word LSTM generates the specific text. The "secret sauce" of this model is Medical Concept Enrichment. The authors extracted 69 frequent medical concepts (e.g., "cardiomegaly", "opacity") from training reports and trained the encoder to predict these. These predicted concepts are then fed into the word-level decoder via a dedicated attention layer, keeping the model grounded in clinical reality.

Experimental Results

The model was pre-trained on the massive CheXpert dataset (224,316 images) to learn domain-specific features before being fine-tuned on the Indiana University Chest X-Ray (IU-RR) benchmark. Results show that domain-specific pre-training is significantly more effective than using standard ImageNet weights.

The proposed model, MvH+AttL+MC (Multi-view Hierarchical + Late Attention + Medical Concepts), outperformed previous state-of-the-art (SOTA) models across all standard NLP metrics:

Metric MM-Att (Previous SOTA) Proposed Model
BLEU-4 0.195 0.255
METEOR 0.274 0.343
ROUGE 0.366 0.453
Qualitative evaluation comparing ground truth with various model configurations.

Qualitative analysis (Figure 6) demonstrates that the inclusion of medical concepts (MC) significantly improves the clinical accuracy of the generated reports. While the base model (MvH+AttL) might produce fluent text, the enriched model better identifies specific findings like "patchy airspace opacities" that align with the ground truth and visual heatmaps.

Conclusion

This research demonstrates that automatic radiology report generation can be significantly improved by treating multi-view images as a complementary pair and explicitly integrating medical concepts into the decoding process. Future work will focus on addressing the limitations of small training corpora and exploring unsupervised pre-training on larger volumes of unpaired medical text to further improve language patterns and clinical robustness.

Summary updated

Abstract

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.
BibTeX
@inproceedings{yuan2019automatic,
  author = {Jianbo Yuan and Haofu Liao and Rui Luo and Jiebo Luo},
  title = {{Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichment}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=c-syXl5QT3u},
}
Figure