Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation

Toru Nishino, Yasuhide Miura, Tomoki Taniguchi, Tomoko Ohkuma, Yuki Suzuki, Shoji Kido, Noriyuki TomiyamaEMNLP 2022 · DOI

Introduction

While most AI radiology tools focus solely on factual accuracy, this paper argues that the order of descriptions is just as critical for clinical utility, especially in multi-phase imaging like contrast CTs. The authors introduce CoPlan, a coordinated planning framework that uses reinforcement learning and a novel inference method to ensure that generated reports are both factually correct and chronologically consistent.

The Challenge of Multi-Phase Imaging

Generating radiology reports automatically is a high-stakes "data-to-text" task. Most existing research focuses on single-phase images (like a standard chest X-ray), where the goal is simply to list findings accurately. However, modern diagnostics often involve multi-phase imaging (e.g., liver contrast CTs), where a patient is scanned multiple times over several minutes.

Comparison of report requirements for plain Chest X-rays versus multi-phase Liver Contrast CTs.

In these cases, a radiologist must describe how a lesion changes over time (e.g., "strong enhancement in the arterial phase, followed by washout in the delayed phase"). If an AI lists these findings out of order, the report becomes confusing or clinically useless. Current "planning-based" models—which first create an outline (a plan) and then write the text—suffer from cascading errors: if the planner makes a mistake in the ordering, the text generator has no way to fix it.

CoPlan: Coordinated Planning Framework

The authors propose CoPlan (Coordinated Planning), a system designed to synchronize the "planner" and the "generator." The architecture consists of three stages: an Image Classifier, a Content Planner (CP), and a Text Generator (TG).

The CoPlan architecture showing coordination through Reinforcement Learning (RL) and Beam Search (BS).

The core innovation of CoPlan lies in two specific coordination strategies:

  1. CoPlan in Reinforcement Learning (RL): Instead of training the planner in isolation, the authors use the quality of the final generated report as a reward signal. They employ a "Report Evaluator" that reconstructs finding labels from the text to check for factual accuracy and correct chronological order.
  2. CoPlan in Beam Search (BS): During the inference (prediction) phase, the system doesn't just pick the most likely plan. It uses a coordinated beam search that looks ahead at what the text generator would produce for a given plan and selects the one that results in the highest-quality final report.

To measure success, they developed two specialized reconstructors: REC_fact (for factual accuracy using a fine-tuned ELECTRA model) and REC_ord (for description order using a fine-tuned T5 model).

Datasets and Evaluation

The study utilizes two distinct datasets to test different scenarios:

  • JLiverCT: A newly introduced dataset of 1,083 Japanese radiology reports of liver lesions. This is "time-series critical," meaning the order of phases (arterial, equilibrium, delayed, etc.) is essential.
  • MIMIC-CXR: A massive, standard English dataset of chest X-rays. While these are "time-series irrelevant," the authors used them to prove that CoPlan also improves factual accuracy in traditional settings.

Key Results

The CoPlan framework showed significant gains across both languages and modalities:

  • Chronological Consistency: On the JLiverCT dataset, CoPlan improved the Content Order (CO) score by 5.1 points compared to baseline models.
  • Factual Accuracy: On the MIMIC-CXR dataset, CoPlan improved the clinical factual accuracy F-score by 9.1 points in scenarios where time-series wasn't a factor.
  • Human Evaluation: Professional radiologists confirmed the results, rating CoPlan higher for both "correctness" and "content order" compared to standard T5 baselines.

Conclusion and Future Work

The authors emphasize that planning-based models are less prone to "hallucinations" than end-to-end models because the generator is strictly constrained by the plan. By bridging the gap between planning and generation, CoPlan addresses the cascading error problem.

However, challenges remain. The coordinated beam search is computationally expensive, taking roughly 10 seconds per report compared to 0.4 seconds for standard models. Additionally, AI still struggles to mimic the nuanced intentions of a radiologist. Future work will focus on "human-in-the-loop" systems where doctors can interactively correct plans before final text generation.

Summary updated

BibTeX
@inproceedings{nishino2022factual,
  author = {Toru Nishino and Yasuhide Miura and Tomoki Taniguchi and Tomoko Ohkuma and Yuki Suzuki and Shoji Kido and Noriyuki Tomiyama},
  title = {{Factual Accuracy is not Enough: Planning Consistent Description Order for Radiology Report Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.480},
}
Figure