Introduction
Medical imaging serves as a cornerstone of modern diagnostic medicine, yet its clinical efficacy is contingent upon rigorous quality control (QC). Currently, tasks such as verifying anatomical positioning in radiographs or identifying logical inconsistencies in radiology reports are performed manually by clinicians. This process is not only labor-intensive and prone to subjective bias but also fails to scale with the increasing volume of medical data. While Large Language Models (LLMs) have demonstrated potential in general medical question-answering, their application in specialized clinical auditing is limited by a lack of standardized benchmarks and a clear understanding of which model architectures—multimodal or reasoning-heavy—are best suited for technical QC.
Methodology: The Hybrid Intelligence Framework
To address these challenges, this study introduces a Hybrid Intelligence Framework centered on a closed-loop evaluation system. The core of this approach is the Human-AI Synergy pipeline, which integrates the expertise of board-certified radiologists with the processing power of AI.
The framework employs a multi-stage process:
- Expert Annotation: Radiologists provide "gold standard" labels for technical errors.
- Multimodal Prompting: For Chest X-rays (CXR), models analyze high-resolution images to detect 11 specific technical issues, such as scapular overlap or truncated pulmonary apices.
- Reasoning-Based Auditing: For CT reports, models act as auditors to identify eight error categories, including interpretive contradictions and metric inconsistencies.
- Closed-Loop Evaluation: The system uses Macro-F1 scores to ensure models are penalized for missing rare but clinically significant errors, such as laterality mislabeling.
Dataset Curation
The researchers developed a specialized benchmark using data from Zhongda Hospital (Southeast University). The dataset consists of:
- 161 Frontal CXR Radiographs: Annotated according to ACR and Fleischner Society guidelines.
- 219 Structured CT Reports: Categorized into major error types ranging from simple typos to complex logical discrepancies.
- De-identification Pipeline: A hybrid approach using Regular Expressions and BioBERT-CRF was implemented to strip Protected Health Information (PHI) while maintaining clinical context.
Results and Discussion
The evaluation revealed significant performance variances across different model architectures. In the CXR quality control task, Gemini 2.0-Flash emerged as the top performer.
As shown in the results, Gemini 2.0-Flash achieved a normalized Macro-F1 of 90, demonstrating a robust ability to generalize across diverse technical error types. In contrast, smaller multimodal models like InternVL2-8B (Macro-F1: 6.26) struggled significantly with the spatial reasoning required for medical positioning tasks.
For CT report auditing, DeepSeek-R1 outperformed its peers with a 62.23% recall rate. A critical finding was the "distillation gap": while the full 671B parameter DeepSeek-R1 model excelled, its distilled variants (32B and 70B) showed marked performance degradation. This suggests that the complex reasoning required for medical auditing is a high-order capability that may be lost during model compression. Furthermore, the study identified a "discovery trade-off," where models like InternLM2.5-7B-chat flagged more potential errors but with lower precision, suggesting a role as a sensitive first-pass filter in clinical workflows.
Conclusion
This research demonstrates that a Human-AI synergy framework can significantly augment medical imaging quality control. While top-tier models like Gemini 2.0-Flash and DeepSeek-R1 show promise in automating technical audits, the high failure rate on "hard" cases (approximately 40%) indicates that medical QC remains a frontier task. Future work will focus on expanding the dataset to other modalities like MRI and Ultrasound and utilizing Reinforcement Learning from Human Feedback (RLHF) to further align model outputs with radiologist preferences.
Summary updated