Generative Large Language Models Trained for Detecting Errors in Radiology Reports

Cong Sun, Kurt Teichman, Yiliang Zhou, Brian Critelli, David Nauheim, Graham Keir, Xindi Wang, Judy Zhong, Adam E. Flanders, George Shih, Yifan PengarXiv 2025 · arXiv · OpenReview

TL;DR

This paper explores how large language models (LLMs) can be fine-tuned to automatically detect errors in radiology reports, a crucial step for improving diagnostic accuracy and patient care. By training models like Llama-3 on a combination of synthetic and real-world radiology reports, they achieved significant improvements in identifying various error types, outperforming baseline models and showing promise for real-world clinical applications.

What Problem Does This Paper Address?

Radiology reports are essential for diagnosing medical conditions, but they are prone to errors. These errors can stem from various sources, including the limitations of speech recognition software used for dictation, the subjective nature of interpretation, and cognitive biases. Even minor errors can lead to misdiagnoses or delayed treatments, with serious consequences for patients. While structured reporting and traditional deep learning methods have been used to address this, they have limitations. This paper aims to bridge the gap by investigating the effectiveness of advanced generative LLMs, specifically fine-tuned for the task of detecting errors in radiology reports, a domain that has been relatively underexplored for these models.

Methods

The researchers developed a two-part dataset for training and evaluating their models. The first part consisted of synthetic radiology reports generated by GPT-4, with half being error-free and the other half containing specific types of errors. The second part comprised real-world error-free reports from the MIMIC-CXR database, augmented with synthetically generated errors by GPT-4.

Four types of errors were focused on:

  • Negation errors: Incorrect use or omission of negative phrases, reversing the meaning.
  • Left/right errors: Mismatched anatomical side labeling.
  • Interval change errors: Alterations in measurements or findings that could lead to misinterpretation.
  • Transcription errors: Typos, misspellings, or incorrect medical terms due to speech recognition issues.

Several LLMs, including Llama-3 (8B and 70B Instruct versions), GPT-4, and BiomedBERT, were evaluated. They employed different strategies: zero-shot prompting (no examples given), few-shot prompting (a few examples provided), and fine-tuning. The performance was measured using F1 scores, confidence intervals, and paired-sample t-tests, with final predictions reviewed by radiologists.

Datasets

  • Synthetic Radiology Reports: 1,656 reports generated by GPT-4, with 828 error-free and 828 containing errors.
  • MIMIC-CXR Database Reports: 614 reports, with 307 error-free and 307 corresponding synthetic reports with errors generated by GPT-4.
  • Real-world Thoracoabdominal Chest X-ray Reports: A dataset of 122,025 reports from heart failure patients was used for a real-world evaluation phase.

The paper does not explicitly introduce a new benchmark dataset but rather constructs a dataset by combining synthetic data with existing real-world data to address the specific task of error detection.

Key Insights and Ideas

  • Fine-tuning is Crucial: The study strongly suggests that fine-tuning LLMs on domain-specific data (radiology reports) significantly enhances their performance in specialized tasks like error detection, outperforming zero-shot approaches.
  • Llama-3 Shines: The fine-tuned Llama-3-70B-Instruct model emerged as the top performer, demonstrating robust capabilities across various error types.
  • Prompting Matters: While zero-shot learning offers good generalizability, few-shot learning can boost performance on specific tasks, but its effectiveness hinges on the quality and relevance of the provided examples.
  • Transcription Errors are a Challenge: While LLMs showed promise, transcription errors, often subtle and context-dependent, still presented a significant challenge, though the models performed well in this area too.

Key Results

  • The fine-tuned Llama-3-70B-Instruct model achieved the highest overall F1 score of 0.780.
  • Specific F1 scores for Llama-3-70B-Instruct (fine-tuned, zero-shot) were:
  • Negation errors: 0.769
    • Left/right errors: 0.772
    • Interval change errors: 0.750
    • Transcription errors: 0.828
  • In a real-world evaluation with 200 reports, the models detected errors that were confirmed by both radiologists in 99 cases (49.5% accuracy) and by at least one radiologist in 163 cases (81.5% accuracy).
  • The fine-tuned Llama-3-70B-Instruct generally outperformed baseline models like BiomedBERT and GPT-4-1106-Preview.

Limitations and Future Work

The authors acknowledge several limitations:

  • Computational Resources: Fine-tuning LLMs requires substantial computational power and high-quality annotated data, which might be a barrier for smaller institutions.
  • Generalizability: The study focused on radiology reports; its applicability to other medical documents or clinical notes is uncertain.
  • Overfitting: There's a potential for overfitting to the specific dataset, which might limit adaptability to different error types or report styles.
  • Synthetic Data Bias: While efforts were made to ensure representative synthetic data, synthetic errors might not fully capture the complexity of real-world errors.

Future work could involve a systematic evaluation of how bias introduced by synthetic errors affects model performance and further validation in diverse clinical settings. The authors also suggest that further real-world validation is needed to assess the models' ability to reduce radiologists' cognitive load and enhance patient care.

Summary updated

Abstract

In this retrospective study, a dataset was constructed with two parts. The first part included 1,656 synthetic chest radiology reports generated by GPT-4 using specified prompts, with 828 being error-free synthetic reports and 828 containing errors. The second part included 614 reports: 307 error-free reports between 2011 and 2016 from the MIMIC-CXR database and 307 corresponding synthetic reports with errors generated by GPT-4 on the basis of these MIMIC-CXR reports and specified prompts. All errors were categorized into four types: negation, left/right, interval change, and transcription errors. Then, several models, including Llama-3, GPT-4, and BiomedBERT, were refined using zero-shot prompting, few-shot prompting, or fine-tuning strategies. Finally, the performance of these models was evaluated using the F1 score, 95\% confidence interval (CI) and paired-sample t-tests on our constructed dataset, with the prediction results further assessed by radiologists. Using zero-shot prompting, the fine-tuned Llama-3-70B-Instruct model achieved the best performance with the following F1 scores: 0.769 for negation errors, 0.772 for left/right errors, 0.750 for interval change errors, 0.828 for transcription errors, and 0.780 overall. In the real-world evaluation phase, two radiologists reviewed 200 randomly selected reports output by the model. Of these, 99 were confirmed to contain errors detected by the models by both radiologists, and 163 were confirmed to contain model-detected errors by at least one radiologist. Generative LLMs, fine-tuned on synthetic and MIMIC-CXR radiology reports, greatly enhanced error detection in radiology reports.
BibTeX
@misc{sun2025generative,
  author = {Cong Sun and Kurt Teichman and Yiliang Zhou and Brian Critelli and David Nauheim and Graham Keir and Xindi Wang and Judy Zhong and Adam E. Flanders and George Shih and Yifan Peng},
  title = {{Generative Large Language Models Trained for Detecting Errors in Radiology Reports}},
  year = {2025},
  eprint = {2504.04336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.04336},
}
Figure