Do LLM Judges Fall for Cytokine-Storm Jargon? An Outcome-Anchored Audit on ICU Sepsis Plans

Language models are increasingly used as automated graders ("LLM-as-a-judge") for medical text, including advice about critically ill patients. We know these graders take shortcuts. They reward length, fake citations, and labels like "EXPERT", and they never admit it in their written reasoning. What we do not know is how badly these shortcuts distort grading of acute clinical decisions, where one wrong call can kill. The main obstacle is that no one has a cheap ground truth there. Human intensivists are scarce, and lexical metrics measure word overlap, not whether a plan is safe.

This note proposes to build that ground truth from an existing sepsis decision model and use it to test one specific, domain-shaped shortcut, which we call mechanism dressing. The question is whether a judge's verdict flips toward a fluid or vasopressor plan when the plan is wrapped in fluent cytokine-storm biology (for example "blocks NF-κB-driven IL-6 transcription" or "limits NETosis-induced capillary leak"), even though the recommended doses are unchanged.

The ground truth comes from ICU-Sepsis: A Benchmark MDP Built from Real Medical Data. It is a Markov decision process, a table of patient states, treatment actions and transition probabilities, estimated from about 17,000 MIMIC-III sepsis admissions. Its 716 states are clusters of 47 clinical features. Its 25 actions are combinations of IV fluid and vasopressor dose. Survival to 90 days is scored 1 and death 0. Solving this table by value iteration gives every state–action pair a value: the estimated survival probability if you take that action now and act optimally afterwards. That value is noisy, but it is computable. It lets us score judges against an outcome-derived signal rather than against another model's opinion.

Core Claim

A judge's ranking of two natural-language resuscitation plans can be scored against the difference in their action values in the ICU-Sepsis model, provided we restrict to states and actions whose transitions are well supported. On such pairs, we hypothesize three things. First, mechanism dressing produces verdict shifts at least as large as the generic authority and provenance cues already documented. Second, the judges' written rationales will not mention the dressing. Third, a judge that first extracts the recommended dose and only then scores it will be much less affected, which would show the shortcut works through how the judge weights content, not through misreading it.

The new element is not "LLM judges in medicine." It is a domain-specific shortcut, measured on an outcome-anchored instrument, with a mechanistic test of where the bias enters.

Relation to Prior Work

Two lines of work motivate the gap without closing it.

Medical judge benchmarks score judges against physician labels.

Both cover outpatient or consumer questions, neither probes shortcut cues, and neither has an outcome-linked reference.

Bias audits measure shortcut cues but outside critical care.

We reuse their measurement protocol: the verdict shift rate (how often a verdict flips when a cue is added) and the cue acknowledgment rate (how often the rationale mentions the cue). Our contribution is a cue that exploits domain knowledge rather than a surface label, and a reference that is not a human preference.

The cytokine-storm literature matters here in a specific way. Papers such as Chrysophanol … NF-κB … sepsis-associated acute kidney injury (Chrysophanol promotes M2 polarization and inhibits M1 polarization through the NF-κB signaling pathway to attenuate sepsis-associated acute kidney injury) and Pyruvate kinase M2 … ferroptosis-associated acute lung injury in cytokine storm supply the vocabulary of mechanisms: NF-κB, M1/M2 macrophage polarization, NETosis, ferroptosis. That vocabulary is plausible-sounding and mostly established in animal models. It is exactly the kind of text a judge might treat as evidence of quality.

The ICU-Sepsis paper also constrains us. It shows that unfiltered rare transitions let agents claim 96% survival, and it imposes a threshold of τ = 20 observed transitions. We inherit that threshold and add a stricter one of our own.

Research Program

Building the test pairs. We solve the ICU-Sepsis model with value iteration to obtain an action value for every state–action pair. We keep only states visited often, and only action pairs where both actions clear the transition threshold and their values differ by a margin larger than a bootstrap confidence interval. The interval comes from resampling the transition counts, so the "better" action is better beyond estimation noise.

Each state's cluster centroid is turned into a clinical vignette using a fixed template: vitals, lactate, creatinine, oxygenation, fluid balance. For each action we write a plan through a controlled template. The template states the dose in words, for example "a moderate 500 mL crystalloid bolus with low-dose norepinephrine," plus a short, neutral rationale. This keeps content identical across variants.

Cue variants. We then create variants that change only the framing:

  • Mechanism dressing: a paragraph of cytokine-storm pathophysiology in favour of the plan, with the doses unchanged.
  • Contradictory dressing: mechanistic text whose logic actually argues against the recommended action, for instance citing capillary leak and lung injury while recommending a large fluid bolus.
  • Fabricated guideline citation: following CALM.
  • Intensivist provenance label: following Silent Judge.
  • Length padding: matched to the token count of the dressing paragraph, so we can separate "longer" from "mechanistic."

Every cue is applied to the worse-valued plan, and separately to the better one, with bidirectional presentation order to cancel position effects.

Judges. - Three to four frontier and open models used as pairwise judges with chain-of-thought. - A rubric version of the Likert judge from the iCliniq study. - An extract-then-score judge: the model first outputs the fluid and vasopressor bin it reads from the plan, then scores that bin given the vignette. Because the extraction step can be checked exactly against the template, it tells us whether dressing makes judges misread doses or merely overweight the prose.

External validity. The model's action values are an estimated, observational quantity, not clinical truth. So an intensivist panel will label 200 pairs, and we report agreement between the value-derived ranking and the panel. We also test whether each judge's mechanism-dressing sensitivity predicts its drop in agreement with doctors on the long-form medical QA benchmark, after we inject the same dressing there.

Baselines

Naive reference points. The first is chance agreement: 50% pairwise accuracy. The second is the policy-level bar in the decision model. ICU-Sepsis reports an expected return of about 0.78 for both random and clinician policies, 0.86 for DQN and PPO, and 0.88 for the optimal policy.

We turn each judge into a policy by letting it pick, in every retained state, the plan it prefers among the candidate actions. We then evaluate that policy's return in the decision model. A judge whose induced policy does not exceed 0.78 on clean plans adds nothing. For dressed plans, the drop from its clean-plan return measures harm in survival units. No reported result was found on Lacuna for any LLM judge or LLM-induced policy on ICU-Sepsis.

Strongest aggregation baseline. This is the BT-σ jury from Who can we trust? LLM-as-a-jury for Comparative Assessment. It fits a Bradley–Terry ranking model that learns a reliability parameter for each judge from pairwise logs, without labels. On SummEval it reports a Spearman correlation of 50.50%, against 45.15% for probability averaging and 49.40% for soft Bradley–Terry. We run it on the same pair logs our individual judges produce, with no extra calls and the same judges.

Beating it means one of two things on dressed pairs:

  • The extract-then-score judge achieves higher accuracy against the value-derived ranking, or a smaller verdict shift rate, with a bootstrap interval that excludes zero.
  • Or, if BT-σ turns out more robust, we report that and recommend it.

Key ablation. The extract-then-score judge versus the same model judging holistically, with identical prompts apart from the extraction step. A second ablation replaces the value-derived reference with the majority verdict of the judges themselves. This tests whether the outcome anchor changes conclusions about which judge is best.

Why not just…?

Why not just use the value table directly and drop the judge? For these templated plans, a lookup of the extracted action would indeed beat any judge, and we will report that as an upper bound. But the table only covers discrete doses in 716 clustered states. Real judges grade free-text plans that include antibiotics, immunomodulators and caveats, and no table covers those. The table is a calibration instrument for judges, not a replacement. The external-validity check on the long-form QA benchmark tests whether what we measure transfers.

Why not just ask intensivists? They are the reference we validate against on 200 pairs. However, the verdict-shift design needs thousands of matched cue/no-cue comparisons per judge, which is infeasible at specialist rates. The long-form QA benchmark itself had to use radiologists and pathologists rather than critical care physicians. If panel agreement with the value-derived ranking is low on high-margin pairs, the anchor fails and we will say so.

Why not just reuse CALM's authority-bias or BiasScope's trap sets? This is a real threat. If a judge's robustness to CALM's fake citations already predicts its mechanism-dressing shift almost perfectly, our cue is redundant. We will compute that correlation across judges and state variants. The contradictory-dressing variant is the part generic benchmarks cannot supply: it needs domain knowledge to notice that the biology argues against the dose.

Why not just use BT-σ? It weights judges by consistency, and a bias shared by all judges is consistent. If every judge prefers dressed plans, BT-σ will confidently rank them higher. Running BT-σ on the same logs will show whether it removes, keeps, or amplifies the shift.

Decisive Checks

The central result is the mechanism-dressing verdict shift rate and cue acknowledgment rate for each judge, on value-anchored high-margin pairs, compared with the length-matched padding control.

The hypothesis fails if:

  • dressing shifts verdicts no more than padding does,
  • or rationales routinely acknowledge the dressing.

The harm result is the drop in judge-induced policy return from clean to dressed plans. A drop that brings a judge below the 0.78 clinician/random level would show the bias is clinically meaningful in this model's terms.

The mechanism result is whether extract-then-score cuts the shift while dose-extraction accuracy stays near perfect. That pattern would locate the bias in how judges weight persuasive prose, not in how they read the plan.