Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Jianbo Yuan, Haofu Liao, Rui Luo, Jiebo Luo

TL;DR

This paper introduces a novel deep learning framework that automatically generates radiology reports from chest X-rays by looking at multiple views (frontal and lateral) simultaneously. By pre-training on massive domain-specific datasets and explicitly "enriching" the model with key medical concepts, the authors achieved state-of-the-art performance in producing accurate, paragraph-length clinical descriptions.

What Problem Does This Paper Address?

Writing radiology reports is a bottleneck in healthcare; it’s time-consuming and requires high-level expertise. While AI has gotten good at "image captioning" (e.g., "a dog in a park"), medical reporting is much harder for three reasons:

  1. Complexity: Medical images contain subtle, overlapping visual cues that must be linked to precise technical language.
  2. Data Scarcity: There are very few high-quality, open-access datasets that pair medical images with full clinical reports.
  3. Precision Requirements: In a standard captioning task, a small error is trivial. In radiology, swapping "left lung" for "right lung" or missing a "pleural effusion" can lead to incorrect treatment.

Existing models often treated different X-ray views as independent or struggled to maintain clinical accuracy over long, multi-sentence paragraphs.

Methods

The authors propose a generative encoder-decoder architecture with several clever modifications:

  • Multi-View Encoder: Using a ResNet-152 backbone, the model processes both frontal and lateral X-rays. To ensure the model doesn't get "confused" by the different angles, they implement a Cross-View Consistency (CVC) loss, forcing the encoder to reach similar diagnostic conclusions regardless of the view.
  • Hierarchical Decoder: Instead of one long sequence, the model uses a two-tiered LSTM. The first tier (Sentence LSTM) decides the "topic" of a sentence, and the second tier (Word LSTM) generates the actual text.
  • Late Fusion Attention: The model doesn't just average the two images. It uses a sentence-level attention mechanism to selectively "look" at the most relevant parts of both the frontal and lateral views for each sentence it writes.
  • Medical Concept Enrichment: This is the "secret sauce." The authors extracted 69 frequent medical concepts (like "cardiomegaly" or "opacity") from training reports. They trained the encoder to predict these concepts and then fed those predictions directly into the word-level decoder via a dedicated attention layer. This acts as a "nudge" to keep the model focused on clinically relevant facts.

Datasets

  • CheXpert: A massive dataset of 224,316 chest X-rays used for pre-training. The authors found that pre-training on medical images is significantly more effective than using standard sets like ImageNet.
  • Indiana University Chest X-Ray (IU-RR): A smaller, standard benchmark (3,074 samples) containing paired images and full reports, used for training the report generation and evaluation.

Key Insights and Ideas

  • Domain-Specific Pre-training is Essential: The jump in performance from ImageNet pre-training to CheXpert pre-training proves that "visual features" in medicine are fundamentally different from those in general photography.
  • Lateral Views Matter: Most previous models treated frontal and lateral images as independent data points. This paper shows that treating them as a complementary pair—and enforcing consistency between them—leads to more robust feature extraction.
  • Explicit Semantics Bridge the Gap: LSTMs often "drift" or hallucinate during long text generation. By explicitly predicting medical concepts and "reminding" the word-decoder of them, the model stays grounded in clinical reality.

Key Results

The proposed model, MvH+AttL+MC (Multi-view Hierarchical + Late Attention + Medical Concepts), outperformed previous state-of-the-art models across standard NLP metrics:

Metric MM-Att (Previous SOTA) Proposed Model
BLEU-4 0.195 0.255
METEOR 0.274 0.343
ROUGE 0.366 0.453

The authors also ran an "Oracle" test (using ground-truth medical concepts), which showed a massive jump in BLEU-4 to 0.303, suggesting that as medical concept detectors improve, report generation will get significantly better.

Limitations and Future Work

  • Small Corpus: The IU-RR dataset is still relatively small, leading to occasional syntax errors or difficulty generating "unseen" sentence structures.
  • Error Propagation: If the encoder misidentifies a concept (e.g., calling a "right lung" a "left lung"), that error is passed directly into the text.
  • Future Directions: The authors suggest exploring unsupervised pre-training on larger volumes of unpaired medical text to help the decoder learn better language patterns without needing perfectly labeled image-report pairs.

Summary updated

Abstract

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.
BibTeX
@inproceedings{yuan2019automatic,
  author = {Jianbo Yuan and Haofu Liao and Rui Luo and Jiebo Luo},
  title = {{Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=c-syXl5QT3u},
}

Loading related content...

Figure