CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation

Ruifeng Yuan, Wanxing Chang, Weiwei Cao, Bowen Shi, Zhongyu Wei, Ling Zhang, Jianpeng Zhang

TL;DR

CT-FineBench is a new question-answering (QA) benchmark designed to evaluate the fine-grained factual consistency of AI-generated Computed Tomography (CT) reports. By shifting the evaluation paradigm from coarse entity matching to a verifiable checklist of specific clinical attributes (like lesion size, location, and margin), it provides a highly interpretable metric that aligns closely with human expert judgment.

What Problem Does This Paper Address?

The automatic generation of radiology reports from CT scans has the potential to significantly optimize clinical workflows. However, deploying these systems safely requires robust evaluation metrics.

Traditional language metrics like BLEU or ROUGE only measure superficial word overlap, often scoring a clinically dangerous report highly simply because it "sounds" professional. Even newer, clinically-aware metrics like RadGraph focus primarily on coarse-grained entity matching (e.g., identifying if a "lung nodule" is present). In radiology, a minor textual error—such as misstating a lesion's size (24mm vs. 4mm) or swapping its location (left vs. right lung)—can completely compromise diagnostic accuracy.

To bridge this gap, the authors address the lack of evaluation tools capable of verifying fine-grained, disease-oriented clinical attributes.

Methods

The authors introduce CT-FineBench, refactoring report evaluation into an automated, fine-grained fact-checking pipeline. The framework operates in two main phases:

  1. Benchmark Construction (Offline):
  2. Attribute Definition: The authors use Named Entity Recognition (NER) and Large Language Models (LLMs) to extract raw (finding, attribute, content) triplets from reference reports. Human annotators then refine these into a structured schema of clinically critical attributes (e.g., mapping "lung nodule" to specific attributes like Location, Shape, Density, and Margin).
  3. QA Pair Construction: Using few-shot prompting with Qwen3-Max, the structured attributes are transformed into a dataset of question-answer pairs grounded in the gold-standard reference reports (e.g., Q: "What is the diameter of the lung nodule?" A: "12 mm").
  4. Evaluation Protocol (Online): Given a machine-generated candidate report, a QA module extracts answers to the benchmark questions. An Answer Comparison module then scores the correctness of these answers (assigning 0, 0.5, or 1.0) using type-aware rules. For instance, numeric attributes are scored based on relative error thresholds, while categorical attributes use synonym-aware exact matching.
The framework of CT-FineBench, illustrating the shift from coarse-grained to fine-grained evaluation and the QA-based protocol.

Datasets

CT-FineBench is built upon two distinct, publicly available CT image-report datasets to cover both chest and abdominal pathologies:

  • CT-RATE: A large-scale chest CT dataset.
  • Merlin: An abdominal CT dataset.

The authors introduce two distinct subsets:

  • CT-FineBench (Evaluation Benchmark): Consists of 44,268 QA pairs derived from 6,646 test reports.
  • CT-FineData (Training Corpus): A massive parallel training corpus containing over 439,665 QA pairs from 44,302 reports, released to help researchers fine-tune and improve report generation models.
Key statistics of the CT-FineBench dataset broken down by source.

Key Insights and Ideas

  • Evaluation as Fact-Checking: Instead of asking an LLM to judge a report holistically (which acts as an uninterpretable black box), decomposing a report into a checklist of atomic, verifiable facts makes evaluation transparent and highly interpretable.
  • The "Lexical Illusion" Exposed: The paper highlights how traditional metrics are easily fooled. In sensitivity tests, lexical metrics gave higher scores to adversarial reports (which contained critical clinical errors but matched the wording of the original) than to paraphrased reports (which were 100% clinically accurate but used different wording).
  • Feasibility of Smaller Evaluators: An ablation study revealed that smaller, open-source models (like Qwen3-32b and Qwen3-8b) perform nearly identically to the frontier Qwen3-Max model when serving as the QA evaluator, making local, cost-effective deployment highly practical.

Key Results

  • High Human Alignment: CT-FineBench achieved the highest correlation with human expert ratings across both datasets. On the CT-RATE dataset, it achieved a Pearson correlation ($\tau$) of 0.622, substantially outperforming the previous state-of-the-art clinical metric, RaTEScore (0.521).
  • Sensitivity to Clinical Errors: In adversarial testing, CT-FineBench was the only metric that behaved as desired—maintaining high scores for paraphrased reports while dropping sharply when fine-grained clinical details were altered.
Figure 3: The heatmap of inter-metric correlation.

Limitations and Future Work

The authors note that CT-FineBench is inherently recall-oriented because the QA pairs are generated strictly from the ground-truth reference reports. This means it excels at catching errors of omission (missing details) but does not explicitly penalize completely fabricated hallucinations that are entirely unrelated to the true findings.

Additionally, the benchmark's scope is currently constrained by the predefined finding labels available in the source datasets. Future work includes expanding the attribute schema to encompass a wider variety of unannotated findings and leveraging the newly released CT-FineData training corpus to directly improve the diagnostic fidelity of report generation models.

Summary updated

Abstract

The evaluation of generated reports remains a critical challenge in Computed Tomography (CT) report generation, due to the large volume of text, the diversity and complexity of findings, and the presence of fine-grained, disease-oriented attributes. Conventional evaluation metrics offer only coarse measures of lexical overlap or entity matching and fail to reflect the granular diagnostic accuracy required for clinical use. To address this gap, we propose CT-FineBench, a benchmark built from CT-RATE and Merlin to evaluate the fine-grained factual consistency of CT reports, constructed from CT-RATE and Merlin. Our benchmark is constructed through a meticulous, Question-Answering (QA) based process: first, we identify and structure key, finding-specific clinical attributes (like location, size, margin). Second, we systematically transform these attributes into a QA dataset, where questions probe for specific clinical details grounded in gold-standard reports. The evaluation protocol for CT-FineBench involves using this QA dataset to query a machine-generated report and scoring the correctness of the answers. This allows for a comprehensive, interpretable, and clinically-relevant assessment, moving beyond superficial lexical overlap to pinpoint specific clinical errors. Experiments show that CT-FineBench correlates better with expert clinical assessment and is substantially more sensitive to fine-grained factual errors than prior metrics.
BibTeX
@inproceedings{yuan2026ct,
  author = {Ruifeng Yuan and Wanxing Chang and Weiwei Cao and Bowen Shi and Zhongyu Wei and Ling Zhang and Jianpeng Zhang},
  title = {{CT-FineBench: A Diagnostic Fidelity Benchmark for Fine-Grained Evaluation of CT Report Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2038},
}
Figure