A Clinical Evaluation of Conformal Prediction for Chest Radiograph Report Generation

Nikolaos Kontogeorgis, Pascal Krezer, Klaus-Rudolf Kladny, Arne Estler, Christian F. Baumgartner, Lisa Koch, Saif Afat, Michael MuehlebacharXiv 2026 · arXiv

TL;DR

Researchers have found that the automated metrics currently used to "guarantee" the safety of AI-generated radiology reports are fundamentally misaligned with actual clinical judgment. To fix this, they introduced CLARITY, a new evaluation framework grounded in radiologist expertise that distinguishes between minor stylistic deviations and clinically dangerous errors.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) are getting better at writing radiology reports, but they still "hallucinate"—they might confidently describe a finding that isn't there or miss a critical one. To deploy these models safely, researchers use a technique called Conformal Prediction (CP).

CP provides a statistical guarantee: it produces a set of possible reports and promises that, with a high probability (e.g., 90%), at least one "admissible" (acceptable) report is in that set. However, the definition of "admissible" usually relies on automated proxies like CheXbert (which extracts clinical labels) or linguistic metrics like BLEU.

The core question this paper tackles is: Do these automated proxies actually reflect what a radiologist considers a safe, admissible report? If the proxy is wrong, the "statistical guarantee" of safety becomes clinically meaningless.

Methods

The authors introduced CLARITY (Clinical Admissibility Criteria), a framework that evaluates reports across 12 clinical categories (e.g., Pneumothorax, Heart silhouette, Pneumonia). Crucially, CLARITY separates accuracy (is the statement true?) from relevance (does this error actually matter for patient management?).

The study workflow involved:

  1. Report Generation: Using a 27-billion parameter model (MedGemma) to generate 10 candidate reports for various chest X-rays.
  2. Expert Annotation: Two board-certified radiologists reviewed 970 generated reports, marking them for correctness and relevance using the CLARITY rubric.
  3. Benchmarking: They compared 14 different automatic metrics against these expert "gold standard" labels.
  4. CP Evaluation: They tested how well Conformal Prediction methods (specifically CLM and SCOPE-Gen) performed when calibrated with automated proxies versus the true clinical labels.
Qualitative comparison showing how traditional metrics (CheXbert) can penalize correct findings if they weren't in the original reference report, whereas CLARITY focuses on image-grounded relevance.

Datasets

  • MIMIC-CXR: A massive, public database of chest radiographs and reports. The authors focused on the test partition, specifically the first study for each of 293 patients to ensure data independence.
  • CLARITY Dataset: A newly introduced expert-annotated benchmark consisting of 97 studies and 970 MLLM-generated reports, which the authors are releasing for reproducible benchmarking.

Key Insights and Ideas

  • The Reference-Based Trap: Most automated metrics compare the AI's report to a "reference" report written by a human. But humans aren't perfect. If the AI correctly identifies a finding that the original radiologist missed, traditional metrics penalize the AI for "hallucinating." CLARITY avoids this by having radiologists judge the AI report directly against the image.
  • Asymmetric Consequences: In medicine, not all errors are equal. Hallucinating a minor bone change is a "minor deviation"; missing a tension pneumothorax is a "consequential error." Automated metrics often treat these as equal, whereas CLARITY's relevance dimension captures this asymmetry.
  • The Proxy Disconnect: The most shocking finding was that CheXbert positive-label F1, a standard metric used in prior safety-critical AI research, showed no positive alignment with actual clinical admissibility. In some tests, it was even weakly negatively correlated.

Key Results

  • Metric Performance: GREEN was the strongest-performing automated metric, with a Spearman correlation of $\rho = 0.700$ with the radiologist endpoint. This is likely because GREEN specifically targets clinically significant errors.
  • The Failure of CheXbert: CheXbert positive F1 had a correlation of $\rho = -0.126$, effectively failing as a safety proxy.
  • CP Trade-offs: When using automated proxies to calibrate Conformal Prediction, the researchers found a "selectivity-coverage trade-off." To reach the desired safety levels (coverage), the models often had to produce impractically large sets of reports, or they simply failed to meet the safety target when evaluated against real clinical standards.
Table 1: Association of automatic metrics with CLARITY. Spearman corre- lations are computed after study-level averaging; 95% confidence intervals use 2,000 study-cluster bootstrap samples and two-sided P values use Holm cor- rection across 14 metric variants. RadCliQ scores are sign-inverted so higher values consistently indicate better reports. The sample-size column gives stud- ies/reports after metric-specific exclusion of undefined values.

Limitations and Future Work

  • Sample Size: The expert-annotated set is relatively small (97 studies), limited by the time board-certified radiologists could spend on the task.
  • Single Model/Dataset: The findings are based on one model (MedGemma) and one dataset (MIMIC-CXR). It's unclear if these results generalize to other modalities (like CT or MRI) or different institutions.
  • Future Directions: The authors suggest that future work should focus on "division of labor"—using automated metrics for large-scale screening while relying on smaller, image-grounded expert datasets to validate that those proxies remain aligned with clinical reality. They also call for multi-institutional studies with more diverse reader populations.

Summary updated

Abstract

While multimodal large language models (MLLMs) have improved automated radiology report generation, they remain prone to hallucinations, necessitating clinically meaningful risk-control methods. Conformal prediction (CP) provides statistical guarantees, yet existing CP approaches rely on uncalibrated automated extractors and fail to account for the clinical relevance of errors. We introduce clinical admissibility criteria (CLARITY), a framework for evaluating radiology reports that explicitly distinguishes clinically consequential errors from minor deviations. To eliminate reliance on automatic judges, we collected radiologist annotations of MLLM-generated reports using CLARITY. By benchmarking CP strategies and automatic evaluation metrics against these expert annotations, we demonstrate that commonly used admissibility proxies systematically fail to detect clinically relevant errors. Our findings reveal that automated admissibility judgments are largely decoupled from true clinical admissibility.
BibTeX
@misc{kontogeorgis2026clinical,
  author = {Nikolaos Kontogeorgis and Pascal Krezer and Klaus-Rudolf Kladny and Arne Estler and Christian F. Baumgartner and Lisa Koch and Saif Afat and Michael Muehlebach},
  title = {{A Clinical Evaluation of Conformal Prediction for Chest Radiograph Report Generation}},
  year = {2026},
  eprint = {2611.10288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10288},
}
Figure