Multimodal Human-AI Synergy for Medical Imaging Quality Control: A Hybrid Intelligence Framework with Adaptive Dataset Curation and Closed-Loop Evaluation

Zhi Qin, Qianhui Gui, Mouxiao Bian, Rui Wang, Hong Ge, Dandan Yao, Ziying Sun, Yuan Zhao, Yu Zhang, Hui Shi, Dongdong Wang, Chenxin Song, Shenghong Ju, Lihao Liu, Junjun He, Jie Xu, Yuan-Cheng WangarXiv 2025 · DOI · arXiv

Introduction

Medical imaging serves as a cornerstone of modern diagnostic medicine, yet its clinical efficacy is contingent upon rigorous quality control (QC). Currently, tasks such as verifying anatomical positioning in radiographs or identifying logical inconsistencies in radiology reports are performed manually by clinicians. This process is not only labor-intensive and prone to subjective bias but also fails to scale with the increasing volume of medical data. While Large Language Models (LLMs) have demonstrated potential in general medical question-answering, their application in specialized clinical auditing is limited by a lack of standardized benchmarks and a clear understanding of which model architectures—multimodal or reasoning-heavy—are best suited for technical QC.

Methodology: The Hybrid Intelligence Framework

To address these challenges, this study introduces a Hybrid Intelligence Framework centered on a closed-loop evaluation system. The core of this approach is the Human-AI Synergy pipeline, which integrates the expertise of board-certified radiologists with the processing power of AI.

The Human-AI Synergy workflow illustrates the multi-stage pipeline from data de-identification to clinical consensus evaluation.

The framework employs a multi-stage process:

  1. Expert Annotation: Radiologists provide "gold standard" labels for technical errors.
  2. Multimodal Prompting: For Chest X-rays (CXR), models analyze high-resolution images to detect 11 specific technical issues, such as scapular overlap or truncated pulmonary apices.
  3. Reasoning-Based Auditing: For CT reports, models act as auditors to identify eight error categories, including interpretive contradictions and metric inconsistencies.
  4. Closed-Loop Evaluation: The system uses Macro-F1 scores to ensure models are penalized for missing rare but clinically significant errors, such as laterality mislabeling.

Dataset Curation

The researchers developed a specialized benchmark using data from Zhongda Hospital (Southeast University). The dataset consists of:

  • 161 Frontal CXR Radiographs: Annotated according to ACR and Fleischner Society guidelines.
  • 219 Structured CT Reports: Categorized into major error types ranging from simple typos to complex logical discrepancies.
  • De-identification Pipeline: A hybrid approach using Regular Expressions and BioBERT-CRF was implemented to strip Protected Health Information (PHI) while maintaining clinical context.

Results and Discussion

The evaluation revealed significant performance variances across different model architectures. In the CXR quality control task, Gemini 2.0-Flash emerged as the top performer.

Normalized Macro F1 scores for CXR quality control across various multimodal models.

As shown in the results, Gemini 2.0-Flash achieved a normalized Macro-F1 of 90, demonstrating a robust ability to generalize across diverse technical error types. In contrast, smaller multimodal models like InternVL2-8B (Macro-F1: 6.26) struggled significantly with the spatial reasoning required for medical positioning tasks.

For CT report auditing, DeepSeek-R1 outperformed its peers with a 62.23% recall rate. A critical finding was the "distillation gap": while the full 671B parameter DeepSeek-R1 model excelled, its distilled variants (32B and 70B) showed marked performance degradation. This suggests that the complex reasoning required for medical auditing is a high-order capability that may be lost during model compression. Furthermore, the study identified a "discovery trade-off," where models like InternLM2.5-7B-chat flagged more potential errors but with lower precision, suggesting a role as a sensitive first-pass filter in clinical workflows.

Conclusion

This research demonstrates that a Human-AI synergy framework can significantly augment medical imaging quality control. While top-tier models like Gemini 2.0-Flash and DeepSeek-R1 show promise in automating technical audits, the high failure rate on "hard" cases (approximately 40%) indicates that medical QC remains a frontier task. Future work will focus on expanding the dataset to other modalities like MRI and Ultrasound and utilizing Reinforcement Learning from Human Feedback (RLHF) to further align model outputs with radiologist preferences.

Summary updated

Abstract

Medical imaging quality control (QC) is essential for accurate diagnosis, yet traditional QC methods remain labor-intensive and subjective. To address this challenge, in this study, we establish a standardized dataset and evaluation framework for medical imaging QC, systematically assessing large language models (LLMs) in image quality assessment and report standardization. Specifically, we first constructed and anonymized a dataset of 161 chest X-ray (CXR) radiographs and 219 CT reports for evaluation. Then, multiple LLMs, including Gemini 2.0-Flash, GPT-4o, and DeepSeek-R1, were evaluated based on recall, precision, and F1 score to detect technical errors and inconsistencies. Experimental results show that Gemini 2.0-Flash achieved a Macro F1 score of 90 in CXR tasks, demonstrating strong generalization but limited fine-grained performance. DeepSeek-R1 excelled in CT report auditing with a 62.23\% recall rate, outperforming other models. However, its distilled variants performed poorly, while InternLM2.5-7B-chat exhibited the highest additional discovery rate, indicating broader but less precise error detection. These findings highlight the potential of LLMs in medical imaging QC, with DeepSeek-R1 and Gemini 2.0-Flash demonstrating superior performance.
BibTeX
@misc{qin2025multimodal,
  author = {Zhi Qin and Qianhui Gui and Mouxiao Bian and Rui Wang and Hong Ge and Dandan Yao and Ziying Sun and Yuan Zhao and Yu Zhang and Hui Shi and Dongdong Wang and Chenxin Song and Shenghong Ju and Lihao Liu and Junjun He and Jie Xu and Yuan-Cheng Wang},
  title = {{Multimodal Human-AI Synergy for Medical Imaging Quality Control: A Hybrid Intelligence Framework with Adaptive Dataset Curation and Closed-Loop Evaluation}},
  year = {2025},
  eprint = {2503.07032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.07032},
}
Figure