Learning to Generate Clinically Coherent Chest X-Ray Reports

Justin R. Lovelace, Bobak MortazaviEMNLP 2020 · OpenReview

Abstract

Justin Lovelace, Bobak Mortazavi. Findings of the Association for Computational Linguistics: EMNLP 2020.

2020.

This paper introduces a Transformer-based model for generating radiology reports that prioritizes clinical accuracy over mere linguistic fluency. By developing a "differentiable" version of the CheXpert labeler and using the Gumbel-Softmax trick, the authors were able to fine-tune their model specifically to reduce clinical errors, outperforming previous recurrent neural network baselines on both standard language metrics and clinical coherence.

Introduction

Radiology is a critical bottleneck in modern healthcare. Radiologists must manually review thousands of images and dictate detailed reports, a process that is time-consuming and prone to fatigue. While automated image captioning has advanced rapidly in general domains, applying it to chest X-rays presents a unique challenge: clinical coherence.

Standard AI models are typically trained to maximize word overlap with ground-truth reports using metrics like BLEU or ROUGE. However, a report can be grammatically perfect while being medically disastrous—for instance, by omitting a critical finding like a pneumothorax. This work aims to bridge the gap between linguistic fluency and medical accuracy.

Methodology

The authors propose a two-stage framework designed to optimize for clinical correctness.

Architecture

The model utilizes a Transformer architecture, departing from traditional Recurrent Neural Networks (RNNs). A pre-trained DenseNet-121 extracts spatial features from the X-ray, which are then processed by a Transformer encoder-decoder to generate the report text.

Differentiable Clinical Labeling

To optimize for clinical accuracy, the model requires a differentiable way to score clinical content during training. The industry-standard CheXpert tool is rule-based and non-differentiable. To overcome this, the authors trained a neural proxy (an LSTM) to mimic CheXpert's behavior with 98.1% F1 accuracy.

The proposed model architecture integrates a differentiable CheXpert proxy to enable fine-tuning via a Clinical Coherence Loss.

Fine-Tuning with Gumbel-Softmax

To backpropagate through discrete word choices, the researchers employed the Gumbel-Softmax trick. This provides a continuous approximation of word sampling, allowing the definition of a "Clinical Coherence Loss" to minimize medical discrepancies during fine-tuning.

Experimental Results

The study utilized the MIMIC-CXR dataset, the largest public repository of paired chest X-rays and reports. The evaluation focused on the "Findings" section of the reports.

Language Generation

The Transformer model set a new benchmark in linguistic quality, achieving a BLEU-4 score of 14.6, outperforming competitive baselines such as "Show, Attend and Tell" (12.8) and "Adaptive Attention" (13.4).

Clinical Coherence

The primary contribution is the significant improvement in clinical accuracy. The fine-tuned Transformer reached a Micro-Avg F1 score of 41.1, a substantial improvement over the standard Transformer (39.8) and the best recurrent baseline (34.7).

Comparison of clinical coherence metrics (F1, Precision, Recall) across different models, highlighting the superior performance of the fine-tuned Transformer.

Statistical analysis using McNemar’s test confirmed that these improvements in clinical coherence were statistically significant ($p < 1 \times 10^{-10}$).

Conclusion

This research demonstrates that clinical accuracy can be improved without sacrificing linguistic fluency by using differentiable neural proxies for domain-specific knowledge. While an F1 score of 41.1 indicates that the model is not yet ready for autonomous clinical use, it represents a major step forward. Future work will focus on addressing rare conditions, expanding the label set beyond the 14 CheXpert observations, and exploring hybrid abstractive-retrieval methods.

Summary updated

Abstract

Justin Lovelace, Bobak Mortazavi. Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.
BibTeX
@inproceedings{lovelace2020learning,
  author = {Justin R. Lovelace and Bobak Mortazavi},
  title = {{Learning to Generate Clinically Coherent Chest X-Ray Reports}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=ZddkmsqFOn0},
}
Figure