Error Detection in Emergency Radiology Reports Using a Large Language Model: A Multistage Evaluation Study (Preprint)

Bin Zhang, Hui Shen, Tianyang Wu, Fei Wang, Jin Fang, Yuange Li, Xiaoling Wu, Shuyi Liu, Liting Chen, Qiuping Ren, Meng Xiao, Jiatong Xu, Jie Sun, Yujie Zhao, Xin Liu, Liaoyuan Wang, Guipeng Mai, Jingjing You, Zhe Jin, Xuewei Wu, Wenle He, Xue Han, Shuixing Zhang, Dong ZengJournal of Medical Internet Research 2025 · DOI

TL;DR

This study introduces DeepSeek-R1, a domain-optimized large language model (LLM) for detecting errors in Chinese emergency radiology reports. It demonstrates that DeepSeek-R1 can achieve performance comparable to experienced radiologists, particularly in identifying critical errors, and significantly reduces reporting time, making it a promising tool for improving quality control in high-pressure clinical settings.

What Problem Does This Paper Address?

Emergency radiology demands high accuracy under tight deadlines. However, increasing workloads and the inherent complexity of emergency cases raise the risk of reporting errors, which can lead to misdiagnoses and delayed treatments. While LLMs have shown promise in general radiology report proofreading, their effectiveness in the specific, time-sensitive, and often non-English context of emergency radiology remains largely unexplored. Existing studies often use synthetic errors or focus on English-language reports, limiting their applicability to real-world Chinese emergency settings. This paper aims to bridge this gap by evaluating a Chinese-language-optimized LLM, DeepSeek-R1, in this critical domain.

Methods

The study employed a four-stage experimental design to rigorously evaluate DeepSeek-R1's performance.

  • Stage 1: Initial Model Screening: Five LLMs were initially screened using 200 error-free reports and 100 reports with synthetically introduced errors across five categories (omission, insertion, spelling, side confusion, other). This stage identified the top-performing models.
  • Stage 2: Few-Shot Prompt Evaluation: The best LLMs were further evaluated using both "0-shot" (no examples) and "few-shot" (6 example reports) prompting techniques on a test set of 100 reports with 64 errors. This aimed to assess if providing examples improved performance.
  • Stage 3: Exploratory Analysis: DeepSeek-R1's performance was benchmarked against 12 radiologists (4 senior, 4 attending, 4 residents) using the same test set from Stage 2, under both 0-shot and few-shot conditions. This stage also included a comparison on a separate dataset (100 reports with 60 errors) from independent institutions.
  • Stage 4: Real-World Validation: The final stage involved evaluating DeepSeek-R1 on 800 unverified real-world emergency radiology reports to assess its practical utility and the clinical impact of its false-positive alerts.

The study meticulously defined error categories and used statistical methods, including Wald chi-square tests and Cohen's kappa, to compare LLM and radiologist performance.

Datasets

  • Dataset 1: Comprised 7435 emergency radiology reports (CT, MRI, radiography) collected from November 2024 to April 2025 from the First Affiliated Hospital of Jinan University. This dataset was used for initial screening and subsequent stages, with subsets created for specific evaluations.
  • Dataset 2: Consisted of 100 error-free and 100 erroneous reports collected from two independent hospitals (Nanfang Hospital and the Affiliated Hospital of Guangdong Medical University) to assess generalizability.
  • Real-World Validation Dataset: 800 unverified emergency radiology reports were used in Stage 4 to assess practical performance.

The study also introduced a benchmark of 100 reports with 64 synthetically introduced errors, designed to stress-test the models and radiologists.

Key Insights and Ideas

  • Domain-Specific LLMs Matter: DeepSeek-R1, optimized for Chinese clinical text, significantly outperformed general LLMs in this specific domain, highlighting the importance of language and domain-specific fine-tuning.
  • Few-Shot Learning Boosts Performance: Providing a few examples (few-shot learning) substantially improved DeepSeek-R1's error detection rate, suggesting that even limited context can guide LLMs more effectively.
  • AI as an Assistive Tool: DeepSeek-R1's performance, particularly in identifying critical omissions, suggests it can act as a valuable assistive tool, augmenting rather than replacing human radiologists.
  • Efficiency Gains are Significant: LLMs demonstrated a dramatic reduction in reading time compared to human readers, a crucial factor in fast-paced emergency settings.
  • Subjectivity in Error Detection: The interrater agreement analysis revealed considerable variability among human readers, underscoring the inherent subjectivity in radiology report interpretation and the potential for AI to offer more standardized assessments.

Key Results

  • High Detection Rate: In a few-shot setting, DeepSeek-R1 achieved an 84.4% error detection rate on a stress-tested dataset, significantly outperforming radiology residents and showing no statistically significant difference compared to senior and attending radiologists.
  • Critical Error Detection: DeepSeek-R1 detected 100% of critical omissions, far exceeding residents' rates (25-50%).
  • Efficiency: DeepSeek-R1 processed reports significantly faster than human readers, with an average reading time of 92 seconds compared to radiologists' 109 seconds.
  • Real-World Utility: In a real-world validation, DeepSeek-R1 identified 117 true reporting errors with a positive predictive value (PPV) of 56.5%. While the PPV is moderate, the efficiency gains are substantial.
  • False Positives: The study noted that false-positive alerts from DeepSeek-R1 were often due to over-sensitivity rather than fabricated findings, and approximately 19% were considered potentially beneficial for improving report clarity.

Limitations and Future Work

  • Synthetic Error Density: The stress-testing phase used a higher density of artificial errors than typically found in real-world reports, meaning the detection rates from stages 2-3 should be interpreted as stress-tested capabilities.
  • Simulated Environment: Exploratory analyses were conducted in a simulated environment without imaging data, which may differ from routine clinical practice.
  • Error Taxonomy Subjectivity: The definitions of error categories and false-positive subtypes involve some degree of subjectivity.
  • Cost-Effectiveness: A formal cost-effectiveness analysis was not conducted.
  • Generalizability: While validation on independent datasets was performed, further validation in diverse clinical settings, languages, and environments with lower baseline error prevalence is needed. Language-specific optimization or models with stronger multilingual pretraining may be required for broader applicability.

Future work could involve further domain-specific fine-tuning of LLMs using emergency medicine-focused corpora to enhance performance in specialized areas like trauma or neurologic imaging.

Summary updated

BibTeX
@article{zhang2025error,
  author = {Bin Zhang and Hui Shen and Tianyang Wu and Fei Wang and Jin Fang and Yuange Li and Xiaoling Wu and Shuyi Liu and Liting Chen and Qiuping Ren and Meng Xiao and Jiatong Xu and Jie Sun and Yujie Zhao and Xin Liu and Liaoyuan Wang and Guipeng Mai and Jingjing You and Zhe Jin and Xuewei Wu and Wenle He and Xue Han and Shuixing Zhang and Dong Zeng},
  title = {{Error Detection in Emergency Radiology Reports Using a Large Language Model: A Multistage Evaluation Study (Preprint)}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/86841},
}
Figure