Weakly Supervised Fine-grained Span-Level Framework for Chinese Radiology Report Quality Assurance

Kaiyu Wang, Lin Mu, Zhiyao Yang, Ximing Li, Xiaotang Zhou, Wanfu Gao, Huimao Zhang

Abstract

Quality Assurance (QA) for radiology reports refers to judging whether the junior reports (written by junior doctors) are qualified. The QA scores of one junior report are given by the senior doctor(s) after reviewing the image and junior report. This process requires intensive labor costs for senior doctors. Additionally, the QA scores may be inaccurate for reasons like diagnosis bias, the ability of senior doctors, and so on. To address this issue, we propose a Span-level Quality Assurance EvaluaTOR (Sqator) to mark QA scores automatically. Unlike the common document-level semantic comparison method, we try to analyze the semantic difference by exploring more fine-grained text spans. Specifically, Sqator measures QA scores by measuring the importance of revised spans between junior and senior reports, and outputs the final QA scores by merging all revised span scores. We evaluate Sqator using a collection of 12,013 radiology reports. Experimental results show that Sqator can achieve competitive QA scores. Moreover, the importance scores of revised spans can be also consistent with the judgments of senior doctors.

1. Introduction

In clinical practice, junior doctors often draft initial radiology reports which are subsequently reviewed, revised, and scored by senior physicians. This Quality Assurance (QA) process is critical for ensuring diagnostic accuracy and training residents. However, it is a labor-intensive and subjective task, prone to human error and fatigue.

Automating this process using Natural Language Processing (NLP) typically involves Semantic Textual Similarity (STS), which compares reports at a document level. However, medical reports are unique: a 99% textual match can still result in an "unqualified" report if the 1% difference involves a critical clinical error, such as swapping "left" for "right." Global semantic models often overlook these localized but vital micro-edits. To address this, we introduce SQATOR (Span-level Quality Assurance EvaluaTOR), a framework that evaluates report quality by analyzing fine-grained, span-level edits.

2. Methodology

SQATOR shifts the evaluation focus from the entire document to specific revised spans. The framework consists of three primary stages: report merging, weakly supervised training, and score aggregation.

The SQATOR framework overview, illustrating the transition from report pairs to span-level classification training.

2.1 Report Merging and Span Extraction

The system first identifies differences between junior (w) and senior (ŵ) reports using a Longest Common Subsequence (LCS) algorithm. It extracts spans that were added, deleted, or revised and merges them into a single sequence (w̃) using BIO (Beginning, Inside, Outside) tags to highlight the edits.

2.2 Weakly Supervised Self-Training

Obtaining span-level labels from senior doctors is prohibitively expensive. SQATOR utilizes a weakly supervised approach, bootstrapping from document-level QA scores. We manually labeled a small subset (99 reports) at the span level. For the remainder of the dataset, document-level scores serve as "pseudo-labels" for individual spans. A text encoder (BERT or T5) combined with a binary classifier is then trained, iteratively refining these pseudo-labels based on model confidence.

2.3 Aggregation

To derive a final QA score for the junior report, the model aggregates individual span scores using either an Average (Ave.) or Minimum (Min.) function. The Minimum function is particularly effective for safety-critical applications, as it ensures that a single critical error can disqualify an entire report.

3. Experimental Evaluation

We evaluated SQATOR on a dataset of 12,013 Chinese radiology reports (X-ray, CT, and MRI) categorized by body part: abdomen, neurology, chest, and a combined "full" set. The dataset reflects real-world imbalance, with 10,290 qualified and 1,723 unqualified reports. The models were tested against several baselines, including document-level BERT/T5 models and Llama3-8B.

4. Results and Discussion

SQATOR consistently outperformed all baselines. On the full dataset, SQATOR-Bert-Min achieved an F1 score of 79.90%, surpassing the best BERT baseline by 6.78 points. While traditional cosine-similarity models struggled with class imbalance—often failing to identify unqualified reports—SQATOR maintained high precision and recall by focusing on clinical significance.

Case studies demonstrating SQATOR's ability to distinguish between stylistic additions (qualified) and critical clinical errors (unqualified).

Qualitative analysis (Figure 4) confirms that the model correctly distinguishes between stylistic revisions (e.g., adding "each lobe of") and critical diagnostic errors (e.g., changing "decreased" to "prolonged" in perfusion studies). Stylistic changes receive high "qualified" scores, while clinical errors receive severe penalties.

5. Conclusion

SQATOR demonstrates that local span-level analysis is superior to global semantic similarity for medical report QA. By leveraging weakly supervised self-training, the framework bridges the gap between document-level labels and the need for fine-grained evaluation. Future work will focus on refining the pseudo-labeling process to reduce noise and adapting the framework for real-time integration into hospital Electronic Health Record systems.

Summary updated

BibTeX
@inproceedings{wang2025weakly,
  author = {Kaiyu Wang and Lin Mu and Zhiyao Yang and Ximing Li and Xiaotang Zhou and Wanfu Gao and Huimao Zhang},
  title = {{Weakly Supervised Fine-grained Span-Level Framework for Chinese Radiology Report Quality Assurance}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  url = {https://openreview.net/forum?id=vXR6QOgUTa},
}
Figure