RadRef: A Reference-Free Metric for Radiology Report Generation

Shelly Schwartz, Alan Ropp, Michael B. Sneider, Nur Yildirim, Cree M. Gaskin, Chirag Agarwal

TL;DR

RadRef is a novel, reference-free metric for evaluating AI-generated radiology reports by comparing report claims directly against the original medical image. By leveraging vision-language models (VLMs) and clinical context, it achieves performance comparable to state-of-the-art metrics that require human-authored "gold standard" reports, making it uniquely suited for real-time clinical monitoring.

What Problem Does This Paper Address?

Evaluating AI in radiology is notoriously difficult. Standard NLP metrics like BLEU or ROUGE track word overlap but fail to capture clinical accuracy. Modern "reference-based" metrics (like RadFact or GREEN) are better, but they require a radiologist-authored report to serve as a ground truth.

This creates two major hurdles:

  1. The Deployment Gap: In a real-world clinical setting, the AI is often used to draft a report for a radiologist. At the moment the draft is generated, no human reference exists yet, leaving no way to automatically check the draft's quality.
  2. The Blind Spot: If both the AI and the human radiologist miss the same finding (an omission), a text-to-text comparison will never flag the error.

RadRef tackles these by removing the need for a human reference report entirely, instead "grounding" the evaluation in the actual radiograph and the clinical indication (the reason the scan was ordered).

Methods

RadRef operates through a four-step pipeline, as illustrated in the worked example in Figure 4:

  1. Image Belief Estimation: A VLM (like MedGemma-27B) probes the radiograph for specific findings. To avoid model bias, it uses "opposed probes"—asking both "Is finding X present?" and "Is the chest free of X?" It also incorporates free-form descriptions to ensure high recall.
  2. Report Parsing: An LLM (GPT-4o-mini) extracts claims from the AI-generated report, categorizing findings as asserted, denied, or unmentioned based on a standard vocabulary (Chest ImaGenome).
  3. Per-Finding Scoring: The metric compares the report's claims against the image-derived beliefs using a Brier score. A key innovation here is the "most-specific rule" (Figure 2): if a model identifies "alveolar hemorrhage," it excludes broader parent terms like "lung opacity" to prevent double-counting.
  4. Clinical Weighting & Aggregation: Findings are weighted by their relevance to the clinical indication (e.g., a cough makes "pneumonia" more relevant than an incidental finding) and the VLM's confidence. The final score is normalized for each radiograph.
RadRef Workflow
Figure 1: The RadRef pipeline maps radiographs, reports, and clinical indications to a unified score.
The Most-Specific Rule
Figure 2: To ensure accuracy, RadRef retains only the most specific finding in a hierarchy.

Datasets

The authors validated RadRef using:

  • ReXVal: A benchmark of 50 MIMIC-CXR studies with 200 reports, each annotated by six radiologists for errors.
  • MIMIC-CXR: Used for a larger evaluation cohort of 500 studies to test robustness and clinical preference.
  • New Reader Study: The authors introduced a new evaluation set of 200 reports (100 AI-generated, 100 human-authored) independently annotated by two board-certified radiologists to specifically track omissions and false findings.

Key Insights and Ideas

  • Omissions are the "Killer App": RadRef’s strongest relative performance is in detecting missed findings. Because it looks at the image, it can "see" what the text forgot to mention—a feat impossible for reference-based metrics.
  • Clinical Context Matters: By weighting findings based on the clinical indication, RadRef aligns more closely with how radiologists actually prioritize information.
  • Reference-Free is Competitive: Surprisingly, RadRef achieves agreement with radiologists that is statistically indistinguishable from top-tier reference-based metrics like GREEN and CRIMSON.
  • Hallucination Detection: The authors also introduced RadRef-FF, a specialized readout that specifically targets "false findings" (hallucinations), providing a more granular error analysis.

Key Results

On the ReXVal benchmark (see Figure 5), RadRef achieved a 73.1% agreement with radiologist rankings of report quality. This is comparable to leading reference-based metrics like GREEN (73.9%) and RadFact (72.6%).

Agreement Table
Table 1: RadRef performs comparably to state-of-the-art reference-based metrics.

In the new reader study focusing on omissions, RadRef attained a Kendall’s $\tau_b$ of 0.295, approaching the inter-reader agreement between two human radiologists (0.386). Furthermore, RadRef-FF (the false-finding readout) showed a strong correlation ($\tau_b = 0.32$) with human-identified hallucinations without ever seeing a reference report.

Limitations and Future Work

While a major step forward, RadRef has acknowledged limitations:

  • Granularity: It currently scores finding-level content but does not yet handle laterality (left vs. right), severity, or temporal changes (comparing to old scans).
  • Vocabulary Constraints: It is limited to findings within its predefined vocabulary (e.g., Chest ImaGenome).
  • VLM Dependency: The metric's accuracy is tethered to the quality of the underlying vision-language model. If the VLM misinterprets the image, the score will be wrong.

Future work involves expanding the vocabulary, improving the calibration of image beliefs, and extending the framework to other imaging modalities beyond chest X-rays.

Summary updated

Abstract

AI-generated radiology reports are usually evaluated by comparison with a radiologist-authored reference, leaving no way to score them when no reference exists. We introduce RadRef, a reference-free metric that scores report claims against image-derived beliefs over a clinical finding vocabulary, weighted by relevance to the clinical indication. On ReXVal, RadRef achieves 73.1% agreement on non-tied same-radiograph report pairs with radiologist rankings, comparable to leading reference-based metrics overall. It is also competitive on false findings and has its clearest relative result on omissions (missed findings). We further evaluate 100 model-generated and 100 radiologist-authored reports from 100 MIMIC-CXR studies, all independently annotated by two board-certified radiologists. On omissions, RadRef attains Kendall's tau-b = 0.295, against 0.386 between the two readers themselves, and no compared metric differs significantly from RadRef on this target. The same image-grounded representation provides a false-finding readout, RadRef-FF, with tau-b = 0.32 on ReXVal and 0.29 in the reader study, without requiring a reference. Controlled perturbations confirm that RadRef depends on the radiograph, report, and clinical indication. RadRef enables image-grounded evaluation before a radiologist-authored reference exists, supporting quality monitoring of report-generation systems in clinical settings, and complements reference-based metrics when one is available.
BibTeX
@misc{schwartz2026radref,
  author = {Shelly Schwartz and Alan Ropp and Michael B. Sneider and Nur Yildirim and Cree M. Gaskin and Chirag Agarwal},
  title = {{RadRef: A Reference-Free Metric for Radiology Report Generation}},
  year = {2026},
  eprint = {2611.13168},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13168},
}
Figure