Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes

Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly, Suranjana Datta, Navya Penubadi, Karl Eric Swanson, Siddharth SharmaarXiv 2026 · arXiv

TL;DR

Researchers have developed a new pipeline for evaluating AI-generated clinical notes by "objectifying" messy clinician complaints into precise, enforceable LLM-judged criteria. By breaking down complex feedback into specific instance-level checks, their system achieves a 0.95 recall in detecting clinically significant errors that standard metrics like ROUGE completely miss.

What Problem Does This Paper Address?

AI "ambient scribes"—systems that listen to a doctor-patient visit and draft a clinical note—are being deployed at scale. However, measuring their quality is notoriously difficult.

Standard lexical metrics like ROUGE (which measures word overlap) are often "blind" to critical errors; a note can be 99% identical to a reference but swap "left" for "right" in a surgical plan, a potentially catastrophic mistake. On the other hand, expert checklists created by committees often fail to reflect what clinicians actually care about in their daily workflow.

Recent attempts to use Large Language Models (LLMs) to judge notes based on user feedback hit a wall: the most frequent clinician complaints (like "misplaced information") are often "composite" and "subjective," leading LLMs to produce unreliable or overly aggressive "yes/no" judgments. Consequently, previous researchers simply deleted these hard-to-enforce criteria. This paper argues that these discarded complaints are exactly where the most value lies.

Methods

The authors introduce a methodology centered on Objectification. Instead of asking an LLM a broad, composite question (e.g., "Is the note disorganized?"), they decompose the complaint into a battery of scoped, determinate decisions:

  1. Logical Decomposition: Breaking a vague complaint like "misplacement" into specific sub-metrics like Leakage (content in the wrong section) and Omission (content missing from the right section).
  2. Directional Sub-categorization: Judging specific section pairs (e.g., "Did a plan sentence leak into the History section?") rather than the whole note.
  3. Instance-Level Judging: The LLM judges individual sentences or case facts rather than giving a macroscopic note-level score.
  4. Anchor-then-Classify: The prompt requires the LLM to first extract quotes from the transcript and note as "proof" before making a final judgment.
  5. Human-in-the-loop Adjudication: A process to resolve disagreements between LLM judges and human raters, which helps refine the prompts and correct human labeling errors.

Datasets

The study utilizes three primary data sources:

  • ACI-bench: A standard public corpus of encounter dialogues and expert notes.
  • ACI-Defect (New): A newly released perturbation benchmark where the authors "planted" 11 types of clinically impactful defects (e.g., swapping patient sex, fabricating exam findings, or moving plan details to history) into ACI-bench notes.
  • Production Data: 1,446 notes from a live ambient scribe system, paired with real-world clinician ratings (1–5 stars) and free-text comments.

Key Insights and Ideas

  • The Enforceability Gap: Previous methods discarded nearly half of user complaints because they weren't "enforceable" by an LLM. The authors show that by "objectifying" these complaints, you can turn them into highly accurate judges.
  • Gold Standards Aren't Gold: In a surprising finding, the authors discovered that the "gold standard" ACI-bench reference notes are riddled with errors. Their clinician review found hallucinations in 32 of 40 notes and omissions in all 40. This highlights the danger of relying solely on reference-based metrics.
  • Discrimination over Recall: Asking an LLM a broad question (e.g., "Is there content in the wrong section?") results in the LLM flagging almost everything (70% false positive rate on clean notes). Breaking the question down into specific section-pair checks (e.g., "A&P into History") drastically improves the system's ability to distinguish a real defect from a stylistic choice.
  • Note-level vs. Instance-level: A single "pass/fail" bit for a note is useless for feedback. Defect counts must be tracked per span or sentence to be meaningful for system improvement.

Key Results

  • Detection Power: The objectified LLM judges localized planted defects with a median recall of 0.95. In contrast, standard metrics like ROUGE and BERTScore changed by less than 0.04, effectively failing to see the errors.
  • Improved Accuracy: Splitting a broad "misplacement" complaint into specific section-pair questions raised the Youden’s J statistic (a measure of informedness) from 0.30 to 0.72.
  • Real-world Correlation: On production notes, the "burden" (the rate of defects found by the judges) fell at every clinician-rating step ($\rho = -0.15$). While the correlation is small, it provides convergent evidence that the judges are tracking clinician dissatisfaction.

Limitations and Future Work

  • Data Source: The production data comes from a single site, and clinician ratings are optional/self-selected, which may introduce bias.
  • Prompt Sensitivity: While the authors tested across multiple models (Gemini, GPT-4, Qwen), LLM judges remain sensitive to prompt engineering.
  • Human Subjectivity: Even with objectification, some clinical documentation remains a matter of "preference" rather than "defect" (e.g., where exactly to put a specific diagnosis).
  • Future Work: The authors suggest that future research should focus on validating these judges across more diverse clinical settings and exploring how to use these instance-level scores to automatically "self-correct" AI-generated notes.

For those interested in the benchmark, the authors have released the ACI-Defect dataset and clinician-reviewed labels on GitHub.

Summary updated

Abstract

Evaluating the quality of AI-generated clinical notes is hard: standard metrics like ROUGE measure lexical similarity to reference notes, while expert checklists encode what committees believe matters. Neither captures what clinicians actually complain about. Recent work has shown that evaluation criteria can be mined from user feedback and filtered to criteria that LLM judges can reliably enforce. We argue that this is where much of the value is lost. The criteria that fail enforceability are disproportionately the ones clinicians raise most often, because high-frequency complaints are stated as several questions glued together. No yes/no judgment can grade such composites as posed. We introduce an objectification step that rewrites such criteria into sets of scoped, determinate decisions. We demonstrate the approach through an evaluation pipeline that mines criteria from complaints, objectifies them, and deploys LLM judges against both planted defects and production notes. Across eleven defects planted into ACI-bench reference notes, objectified judges localize violations with median recall of 0.95, while reference metrics change by at most 0.037. Given the complaint as one broad question, the same judges still find every planted defect, but they also flag 70% of the unperturbed reference notes; one section-pair question per call raises Youden's J from 0.30 to 0.72 (by 0.19 to 0.26 with prompt effort matched). We release ACI-Defect, a perturbation benchmark; clinician-reviewed instance-level labels showing quality gaps in the ACI-bench gold standard (32 of 40 notes contain hallucinations; 40 of 40 omit discussed facts); and grounding-judge verdicts. On 1,446 production notes, mean instance-rate burden falls at every clinician-rating step (rho=-0.15), a small association. Defect counts have to be kept per span rather than collapsed into one pass/fail bit per note.
BibTeX
@misc{gupta2026objectify,
  author = {Nikita Gupta and Narayanan Unny Edakunni and Vikram Okaly and Suranjana Datta and Navya Penubadi and Karl Eric Swanson and Siddharth Sharma},
  title = {{Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical Notes}},
  year = {2026},
  eprint = {2611.19525},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19525},
}
Figure