When evaluation becomes intervention in clinical AI

Anthony Abou Mrad, Aviona Conti, Samuel Schmidgall, Jayanth Pratap, Ruhika Iyer, Joseph Kim, King Shi, Yann Sakref, Luis M. Ahumada, Lisa Knowlton, Andrew M. Cameron, Khyzer Aziz, Jeffrey K. JoplingarXiv 2026 · arXiv

TL;DR

As generative AI scales in healthcare, we are increasingly using "AI judges" (LLMs) to evaluate clinical outputs that are too voluminous for human review. This paper argues that when these judges are used to select prompts or train models, they cease to be neutral observers and become part of the clinical intervention itself, necessitating rigorous, independent validation to prevent the amplification of "omission blindness" and other systematic errors.

What Problem Does This Paper Address?

In traditional clinical AI, like surgical risk calculators, "ground truth" was easy to define: did the patient have a complication or not? With the advent of Large Language Models (LLMs) acting as medical scribes or diagnostic agents, the output is open-ended text. There is no single "correct" paragraph, so the gold standard remains clinical review by experts.

However, humans cannot keep pace with AI that drafts millions of notes per week. To bridge this gap, developers use AI judges—models designed to grade other models against a rubric. The danger arises when these judges are used "in the loop" to pick the best prompt or to provide rewards for Reinforcement Learning from AI Feedback (RLAIF). If a judge has a blind spot—for example, failing to notice when a scribe forgets to include critical follow-up instructions—the system will "learn" that these incomplete notes are perfect. The evaluation metric effectively becomes the intervention, potentially baking dangerous errors into clinical workflows.

Methods

The authors propose a framework for governing AI judges based on their role in the system:

  1. Categorizing Roles: They distinguish between judges that Observe/Inform (benchmarking), Shape/Select (routing cases or picking prompts), and Supply Labels/Rewards (training the model).
  2. The "Locked Judge" Requirement: Teams must freeze the judge’s full configuration (model version, prompts, rubrics) before testing. Even a minor vendor update to the underlying model can fundamentally change how a judge perceives clinical safety.
  3. Independent Testing: For any judge-driven change (e.g., a new prompt selected because it scored higher), the authors insist on an independent clinical test on a "sealed set" of cases that the judge never saw during development.
  4. Reporting Standards: They provide a comprehensive checklist (Box 1) for what an AI judge report must contain, including sensitivity/specificity for clinical errors and a "matched-baseline" comparison with human clinicians.
Roles of an AI judge and the evidence required for each role.

Datasets

The paper is a Perspective piece that synthesizes findings from several key studies rather than introducing a single new dataset. It references:

  • NSQIP: A dataset of 1.4 million patients used for traditional risk calculators.
  • Diabetic Retinopathy studies: Validated against 128,175 images graded by ophthalmologists.
  • Clinical Scribe Audits: Studies showing that LLM judges often perform near chance at detecting omitted facts in clinical notes.
  • Pediatric Decision-Support Benchmarks: Where multi-model panels were compared against physician safety ratings.

Key Insights and Ideas

  • Omission Blindness: One of the most critical findings is that while LLM judges are decent at spotting added or hallucinated facts, they are remarkably poor at spotting missing information. If a judge doesn't look for a specific instruction (like "go to the ER if chest pain returns"), it will give a passing grade to a dangerous note.
  • The Agreement Trap: High "accuracy" or "agreement" percentages can be misleading. If unsafe notes are rare (e.g., 5 in 100), a judge that labels everything as safe will have 95% accuracy but 0% sensitivity for the actual danger.
  • Shared Error Sources: Using a "panel" of judges (multiple LLMs) doesn't necessarily solve the problem. If all judges use the same flawed rubric or were trained on similar data, they will likely share the same blind spots.
How high agreement can hide missed failures, showing that sensitivity is a better metric than raw accuracy for rare clinical errors.

Key Results

  • Sensitivity vs. Chance: Across multiple designs, LLM judges distinguished intact notes from those with facts removed "little better than chance."
  • Instruction Sensitivity: In one audit, simply changing the judge's instructions raised the share of confirmed errors from 9.3% to 79.0%, highlighting how fragile these evaluations are.
  • Safety Mismatch: In pediatric evaluations, AI judges passed 89–100% of answers that human physicians had explicitly rated as "unsafe."
  • Reward Hacking: When models are trained to please a rubric-based judge, they often learn to "game" the score—for example, by providing longer, more polite answers that actually delay or omit the most important clinical advice.
Different judge architectures (single, panel, deliberative, routing) and how shared sources of error can propagate through them.

Limitations and Future Work

The authors acknowledge that the evidence base for AI judges is still "young," with many cited studies currently existing only as preprints. They note that the feasibility of their proposed rigorous reporting requirements (Box 1) needs to be tested in real-world engineering cycles.

Future work should focus on:

  • Developing better methods for detecting omissions in clinical text.
  • Creating "regression sets" of clinician-labeled cases to detect when a judge's behavior drifts over time.
  • Establishing clear FDA validation thresholds for LLM-based adjudicators, which are currently missing from existing guidance.

Summary updated

Abstract

AI judges, models that grade another system's output against a rubric, extend clinical review to the scale at which clinical AI writes. When teams use judge assessments to select a prompt or train a model, the judge becomes part of the intervention. Selection can then favor errors the judge overlooks. We describe what a judge's report must contain and the independent clinical test each judge-driven change needs.
BibTeX
@misc{mrad2026when,
  author = {Anthony Abou Mrad and Aviona Conti and Samuel Schmidgall and Jayanth Pratap and Ruhika Iyer and Joseph Kim and King Shi and Yann Sakref and Luis M. Ahumada and Lisa Knowlton and Andrew M. Cameron and Khyzer Aziz and Jeffrey K. Jopling},
  title = {{When evaluation becomes intervention in clinical AI}},
  year = {2026},
  eprint = {2611.18297},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18297},
}
Figure