Do LLM Judges Stop at Viral Suppression? Auditing Evaluators on Discordant Clinical Evidence

In HIV care, combination antiretroviral therapy usually drives the virus in the blood to undetectable levels. Yet a sizeable minority of patients, called immunological non-responders, never rebuild their CD4+ T cells, the immune cells the virus depletes. They stay at elevated risk of death from non-AIDS causes even though the headline marker says treatment worked. The clinical failure here is stopping at the first marker that looks good. This proposal asks whether LLM judges fail the same way. An LLM judge is a large language model prompted to grade other models' answers, and it is now used both to rank medical models and to supply rewards when training them. The worry is that when one salient criterion is satisfied, such as "viral load undetectable, adherence confirmed," the judge rates an answer as correct and never checks the patient's lab values that contradict that conclusion.

The core tool is a counterfactual audit. We hold the candidate answer fixed and change only the secondary immune markers in the patient case, for example CD4 count, CD4/CD8 ratio, age, or years on therapy. We then measure whether the judge's score moves. A judge that actually reads the evidence should punish a reassuring answer once the CD4 count drops from 650 to 180 cells/µL while viral load stays suppressed. A judge that grades the answer's surface should not. We then test one targeted fix: rubrics generated from the case's own lab values rather than from reference texts. We ask whether this fix closes the gap where generic physician checklists and reference-mined rubrics do not.

Core Claim

Our hypothesis is that current LLM judges, including checklist-guided ones, are largely insensitive to discordant evidence in the case context. They reward answers that correctly address the primary endpoint and ignore a stalled secondary endpoint. We further claim that this insensitivity is measurable with lab-panel counterfactuals, and that it is not fixed by stronger models or position swapping. We expect it to be substantially reduced only when the judge is handed rubric items derived deterministically from which markers are discordant in that specific case.

The immunology is not decoration. It provides a domain where discordance has well-characterized clinical thresholds and documented mortality consequences, which makes "correct" answers checkable. Recovery of CD4 counts under therapy plateaus after roughly four years, and the height of that plateau depends on damage sustained before treatment started (Luz et al.). Children tend to plateau near 80% of healthy peers (Picat et al.). Nearly 30% of older virally suppressed patients fail to reconstitute, with roughly tripled non-AIDS mortality (Fan et al. (2023)). Persistently inverted CD4/CD8 ratios predict non-AIDS malignancies even after counts recover (PLoS (2016)). Normal cell counts can also hide lost function, as with regulatory B cells in Graves' disease that return in number but not in suppressive capacity (Zha et al.). Each of these findings defines a case where "the main number looks fine" and "the patient is fine" come apart.

Relation to Prior Work

Three medical judge efforts motivate the work, and none tests this failure. A Benchmark for Long-Form Medical Question Answering collects 1,077 consumer questions with physician pairwise verdicts and compares GPT-4o and Claude judges against those verdicts. It reports severe positional bias, but its items are single-turn and contain no conflicting lab trajectories. LLMEval-Med builds a GPT-4o judge with physician-refined checklists over 2,996 electronic-health-record-derived questions. It finds reasoning and generation scores below 50% even for top models, with DeepSeek-R1 at 64.23% overall. Its checklists are written once per question and are never stress-tested by perturbing the case evidence. MediX-R1 constrains this proposal: it shows that an LLM judge used alone as a training reward is hacked, and it needs embedding and format anchors. That is direct evidence that judge blind spots matter once judges drive optimization.

Two lines of judge research supply method and a second axis. QuRL mines case-wise rubrics from reference texts and filters out rubric items that do not separate answers. It raises human correlation from Spearman ρ = 0.08 for a plain judge to 0.31. We reuse its idea of case-specific rubrics but change the source. Our rubric items come from structured lab values through clinical threshold rules, not from reference prose that may itself omit the discordance. The Silent Judge shows that judges shift verdicts in response to provenance labels (+18% for "EXPERT" on ELI5) and recency labels (up to +30% for GPT-4o). It also shows a 0% rate at which the judge's rationale acknowledges the cue. We borrow its verdict-shift and acknowledgment measures to ask a mirror-image question: does the judge fail to respond to cues it should use, and does its rationale pretend it checked them?

Research Program

The first component is a discordance case bank. We will write around 300 base vignettes spanning five templates. The first two are an adult HIV non-responder (HIV-1 RNA under 50 copies/mL with CD4 under 350 cells/µL after several years of therapy) and an older-adult non-responder. The remaining three are a pediatric patient below the expected ceiling, a patient with recovered CD4 count but inverted CD4/CD8 ratio, and a Graves' remission case with normal regulatory B cell counts but lost suppressive function. The Graves' template tests transfer beyond HIV. Each vignette is stored as structured fields plus narrative text. Every vignette gets a concordant twin in which the secondary markers are normal and everything else is identical.

The second component is a set of controlled candidate answers per vignette. The first is a complacent answer that correctly praises viral suppression and recommends routine follow-up. The second is a discordance-aware answer that flags incomplete immune recovery, its mortality implications, and appropriate workup. The third is a padded complacent answer, made longer and more confident, to separate this failure from length bias. The fourth is a keyword-stuffed complacent answer that mentions "CD4" without drawing the right conclusion, to catch judges that match terms. Two infectious-disease physicians will label each vignette–answer pair for clinical adequacy, blind to condition, on a subset large enough to estimate agreement. The complacent answer is adequate for the concordant twin and inadequate for the discordant one, which is what makes the counterfactual design work.

The third component is the proposed judge. A small rule module reads the structured fields and emits the discordant markers along with required rubric items, such as "states that CD4 recovery is incomplete despite suppression" and "addresses elevated non-AIDS risk given age." The rule thresholds come from the cited cohort literature. The judge must verify each item against the answer before scoring. The design is deliberately simple, so a negative result is informative.

The fourth component is a practical-consequence test. We use each judge to pick the best of eight sampled answers from an open model on held-out vignettes, then have physicians rate the selected answers. We use best-of-n selection rather than full reinforcement learning so that differences come from the judge alone rather than from optimization dynamics.

Baselines

The naive reference point is a plain LLM judge that receives the vignette and answer with a generic 1–5 correctness and helpfulness prompt, run with position swapping. This is the "standard LLM-as-judge" setting whose human correlation QuRL reports as ρ = 0.08. We run it with GPT-4o, an open judge such as Qwen3-14B as used by MediX-R1, and one reasoning model. Including the reasoning model tests whether more deliberation alone fixes the problem.

The strongest existing clinical approach is an LLMEval-Med-style judge that uses per-question physician checklists. We give physicians the same annotation time per vignette that our rule module saves them. They write checklists from the vignette text as LLMEval-Med did, and the checklists are refined once against a calibration split. On its own benchmark, the reported number for this family is the model score (DeepSeek-R1 at 64.23% overall), not a judge–physician agreement figure. For discordant immune cases specifically, there is no reported result on Lacuna, so we measure it.

The strongest rubric-generation approach is QuRL-style mining. Rubrics are extracted from a reference answer written by a physician for each vignette, then variance-filtered as in QuRL, with the same judge model and token budget as ours. Its reported gain is ρ = 0.31 against 0.08 on open-domain long-form writing.

The key ablation keeps our judge prompt and verification step but replaces the lab-derived rubric items with the union of all template rubric items, applied regardless of which markers are discordant. This isolates whether the benefit comes from conditioning on the case's actual values or merely from mentioning immune-recovery criteria. All judges share model, temperature, swap protocol, and calibration split. Beating the baselines means two things: higher counterfactual sensitivity with a paired bootstrap interval excluding zero, and physician agreement at least as high as the checklist judge on concordant twins, so that we have not simply made the judge harsher.

Why not just…?

Why not just have physicians write a checklist, as LLMEval-Med does? This is the most serious threat. A physician reading an INR vignette will write "addresses low CD4," and that might suffice. The counterfactual design settles it. Checklists written against the discordant vignette may still be applied by a judge that ticks the item whenever "CD4" appears, which the keyword-stuffed answer tests. Checklists written once per question also cannot follow the concordant/discordant flip unless they are rewritten for each variant. If physician checklists match our judge on sensitivity and keyword robustness, the honest conclusion is that the contribution is the audit, not the rule module, and we will report it that way.

Why not just mine rubrics from reference answers, as QuRL does? Reference-mined rubrics inherit whatever the reference emphasizes. In a domain where, as the file's gap analysis argues, suppression is routinely conflated with recovery, a reference may underweight the stalled marker, and variance filtering keeps items that separate answers rather than items that are clinically decisive. The QuRL-style baseline, run with physician references, directly measures this. If it performs equally well, the rule module is unnecessary.

Why not just use a stronger reasoning model? It is plausible that deliberation surfaces the contradiction. Evidence against relying on it comes from The Silent Judge: stated reasoning did not reflect what drove verdicts (0% acknowledgment), so fluent rationales are not proof that the evidence was read. The reasoning-model baseline, scored by counterfactual sensitivity rather than rationale content, tests this empirically.

Why not just use physician panels? They are the gold standard and serve as ground truth here. But best-of-n selection and training loops need thousands of judgments, and MediX-R1 shows that automated judges are already being placed in those loops. Knowing where they fail is therefore necessary regardless.

Proposed Experiments and Decisive Checks

The primary metric is counterfactual sensitivity. For a fixed complacent answer $a$ and vignette pair $(c^{+}, c^{-})$, where $c^{+}$ is the concordant case and $c^{-}$ the discordant one, it is defined as

$$\Delta = \mathbb{E}\left[s(a, c^{+}) - s(a, c^{-})\right],$$

where $s$ is the judge's normalized score and the expectation runs over vignettes. It is reported alongside the same gap computed from physician labels, which serves as the target. Secondary metrics are pairwise accuracy against physicians on discordant cases, Spearman correlation with physician scores, and verdict shift under the padded and keyword-stuffed answers. We also track the acknowledgment rate, meaning how often the rationale cites the secondary markers, compared against whether the score actually moved.

The decisive check is the gap between acknowledgment and sensitivity. If judges mention CD4 in their rationales while $\Delta$ stays near zero, the silent-judge phenomenon extends to ignoring evidence the judge should use, a new and worrying result for clinical auditing.

A second check tests shortcut interaction. We attach "2025 guideline" or "expert-authored" labels to the complacent answer, following the Silent Judge protocol, and ask whether authority cues further suppress sensitivity. A third check guards against over-flagging: we run all judges on LLMEval-Med items and the Lavita set from Hosseini et al. and verify that agreement with the released physician verdicts does not degrade. The fourth check is the best-of-n study, where physician ratings of selected answers show whether evaluator blind spots translate into worse chosen outputs.

The proposal fails cleanly if the plain judges already show $\Delta$ close to the physician gap. That would itself be a useful negative result for the medical-judge literature.