On the Automatic Generation of Medical Imaging Reports

Baoyu Jing, Pengtao Xie, Eric P. Xing

Abstract

Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time-consuming and tedious for experienced physicians. To address these issues, we study the automatic generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings and tags. Second, abnormal regions in medical images are difficult to identify. Third, the reports are typically long, containing multiple sentences. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on two publicly available datasets.

Introduction

Writing medical imaging reports is a critical yet burdensome bottleneck in modern healthcare. For experienced radiologists, the process is often tedious and time-consuming; for trainees, it is demanding and prone to error. Automating this process requires moving beyond standard image captioning—which typically describes simple scenes like "a cat on a mat"—to address the unique complexities of clinical data.

Medical reports present three primary challenges:

  1. Heterogeneity: Reports combine high-level clinical tags (keywords) with descriptive findings (paragraphs).
  2. Localization: Pathological abnormalities often occupy only tiny fractions of an image, making them easy for general models to overlook.
  3. Narrative Length: Unlike single-sentence captions, medical reports are long, structured paragraphs that require long-term coherence.

Methodology

To address these hurdles, the authors introduce a multi-task learning framework designed to generate long, coherent medical reports from images such as chest X-rays.

Overview of the proposed multi-task learning framework for medical report generation.

Multi-Task Learning

Instead of focusing solely on text generation, the model is trained to simultaneously predict clinical tags (multi-label classification) and generate report text. This dual objective forces the model to learn more robust internal representations of medical content.

Co-Attention Mechanism

The "secret sauce" of this architecture is the co-attention mechanism. While standard models use visual attention to scan image regions, this model simultaneously weighs visual features (pixels) and semantic features (embeddings of predicted tags). This allows the predicted tags to act as "semantic anchors," guiding the visual system to "zoom in" on relevant regions, such as the pleural space when a "pneumothorax" tag is predicted.

Hierarchical LSTM

To maintain coherence over long sequences, the model employs a two-tiered LSTM structure. A Sentence LSTM determines the "topic" of the next sentence and manages the stop control. A subsequent Word LSTM then translates that topic into a specific sequence of words, mirroring the way humans structure reports by addressing one anatomical region or finding at a time.

Experimental Setup

The model was evaluated on two key datasets:

  • IU X-Ray: A collection from Indiana University featuring 7,470 chest X-ray pairs with corresponding findings, impressions, and tags.
  • PEIR Gross: A pathology image library with 7,442 image-caption pairs, used to test the model's ability to generate concise descriptions across various sub-categories.

Results and Discussion

The proposed Ours-CoAttention model consistently outperformed state-of-the-art baselines, including standard CNN-RNN and Soft Attention models, across all major metrics (BLEU, METEOR, ROUGE, and CIDEr).

Quantitative Performance

On the IU X-Ray dataset, the model achieved a BLEU-4 score of 0.247, nearly doubling the performance of the standard Soft-ATT image captioning model (0.118). This significant margin highlights the importance of the hierarchical and semantic components.

Qualitative Localization

Qualitatively, the model demonstrated a remarkable ability to localize abnormalities. By attending to specific regions corresponding to predicted tags, it correctly identified and described complex findings such as "calcified granulomas" and "degenerative changes of the spine."

Conclusion and Future Work

While the results represent a significant step forward, the authors identified areas for improvement. The model remains sensitive to image quality (e.g., contrast and brightness) and can occasionally struggle with fine-grained accuracy—for instance, correctly identifying the location of an abnormality but mislabeling its specific type (e.g., calling an "eventration" an "interstitial opacity").

Future research will focus on improving the tag prediction module, incorporating more diverse datasets, and potentially integrating patient history as an additional input to the co-attention mechanism to further enhance clinical utility.

Summary updated

Abstract

Medical imaging is widely used in clinical practice for diagnosis and treatment. Report-writing can be error-prone for unexperienced physicians, and time- consuming and tedious for experienced physicians. To address these issues, we study the automatic generation of medical imaging reports. This task presents several challenges. First, a complete report contains multiple heterogeneous forms of information, including findings and tags. Second, abnormal regions in medical images are difficult to identify. Third, the re- ports are typically long, containing multiple sentences. To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the pre- diction of tags and the generation of para- graphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs. We demonstrate the effectiveness of the proposed methods on two publicly available datasets.
BibTeX
@inproceedings{jing2018automatic,
  author = {Baoyu Jing and Pengtao Xie and Eric P. Xing},
  title = {{On the Automatic Generation of Medical Imaging Reports}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=KmE55odhfx7},
}
Figure