Where Does an LLM Judge's Bias Travel? Separating Rationale-Mediated from Direct Effects

When a large language model is used as a judge to grade or compare other models' answers, it usually writes a justification first and then a verdict. Recent work shows these judges flip their verdicts in response to irrelevant cues, such as a fake date or an "EXPERT" author label, while their written justifications never mention the cue. This leaves a basic question open. Is the bias hidden inside the justification, as a harsher tone or selectively noticed flaws that then drive the verdict? Or does the verdict bypass the justification and read the cue directly? These are different failures with different fixes. They also have different consequences for anyone who audits judges by reading their rationales.

We propose to answer the question with a borrowed tool from causal fairness: mediation analysis, which splits a cause's total effect into the part that flows through an intermediate variable and the part that does not. For an LLM judge, the intermediate variable is the written rationale. Here the usual weakness of causal fairness becomes a strength. In tabular settings, counterfactual worlds can never be observed and must be assumed. For a judge, we can literally run both worlds. We write the rationale under one cue value, then transplant it into a prompt carrying the other cue value, and read off the verdict. The core claim is that this "rationale transplant" exactly identifies the direct and rationale-mediated parts of judge bias, with no untestable graph assumptions beyond the judge's own computation. Measuring both parts tells us whether rationale-based auditing is trustworthy, which mitigation will work, and whether direct and mediated biases cancel to hide discrimination.

Core Claim

Treat a judge as a two-step program. It reads input $(X, A)$, where $X$ is the content to be judged and $A$ is a cue or protected attribute. It samples a rationale $R \sim p(R \mid X, A)$, then a verdict $\hat{Y} \sim p(\hat{Y} \mid X, A, R)$. Write $R(a)$ for a rationale sampled under $A=a$, and $\hat{Y}(a, r)$ for the verdict given cue $a$ and a fixed rationale $r$. The two quantities of interest are the natural direct effect (NDE) and the natural indirect effect (NIE):

$$\text{NDE} = \mathbb{E}[\hat{Y}(a', R(a))] - \mathbb{E}[\hat{Y}(a, R(a))], \qquad \text{NIE} = \mathbb{E}[\hat{Y}(a, R(a'))] - \mathbb{E}[\hat{Y}(a, R(a))].$$

The NDE changes only what the verdict step sees about the cue, holding the rationale fixed. The NIE changes only the rationale, holding the verdict step's view of the cue fixed. Both are expectations over sampled rationales and verdicts. In ordinary causal fairness, the cross-world term $\hat{Y}(a', R(a))$ is unobservable. Here it is a forward pass on a spliced prompt.

These are the text-judge analogues of the counterfactual direct and indirect effects in Fairness in Decision-Making - The Causal Explanation Formula, which decomposes disparity in tabular decisions and shows that opposing direct and indirect effects can produce zero total disparity. That paper supplies the method. We move it to a setting where the mediator is a full generated text and identification is exact rather than assumed.

The work is motivated by The Silent Judge. That study found verdict shift rates up to +30% for recency cues on ELI5 with GPT-4o, together with a 0% Cue Acknowledgment Rate. A rationale that never names the cue is consistent with either a large NIE or a large NDE, and that study cannot tell them apart. We can.

Research Program

Stage 1: build the estimator. The estimator works with any judge that accepts a prefilled assistant turn or a prior-message rationale. For each item, we sample $k$ rationales under each cue value. We then compute verdict probabilities under all four cue-by-rationale combinations, using the verdict token distribution where logits are available and repeated sampling otherwise. The interaction term is reported alongside NDE and NIE so the decomposition is not forced to add up.

Two controls guard against artifacts. The first is a noise floor: transplant a rationale resampled under the same cue. The second is a coherence check: transplanted prompts can be slightly off-distribution, because a rationale written under "OLD (1950)" now sits next to "NEW (2025)". We measure the judge's perplexity on the rationale in both contexts and report effects separately for low- and high-mismatch items. We also add a paraphrase arm, which runs the rationale through a neutral rewriter to strip any verbatim echo of context.

Stage 2: validate on judges whose mechanism we planted. We fine-tune a small open-weight judge into two variants. One flips verdicts on a cue while writing cue-independent rationales, a pure direct path. The other writes systematically harsher critiques under the cue and then follows its critique faithfully, a pure indirect path. We also build a third variant where the two paths act in opposite directions and roughly cancel. The estimator must recover each planted structure, including the cancellation case. That case is the "zero disparity" trap, and here it shows up as a near-zero verdict shift rate that hides two real effects.

Stage 3: decompose real judges on two kinds of attribute.

Explicit metadata cues. These are the provenance and recency cues from The Silent Judge, plus authority (fabricated citations) and verbosity perturbations from Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. That work measures how often verdicts survive such perturbations but not through which path they act.

Attributes entangled in the text itself. These are author identity references and dialect, which cannot simply be deleted. For these, $A$ acts through a counterfactual rewrite $X(a) \to X(a')$, generated with the LLM rewriting method of Flexible text generation for counterfactual fairness probing. That paper used rewrites to audit a toxicity classifier, never a judge. The direct effect is then "text features of the rewrite that reach the verdict without changing the rationale."

The entangled case also bears on a puzzle in BiasScope. Its automated search, even when seeded with social stereotypes, mostly surfaced cognitive biases. This contrasts with pervasive stereotyping in BBQ. One hypothesis we can test directly: social bias in judges is small in total effect but nonzero in opposing direct and indirect components that partly cancel.

Stage 4: show the diagnosis picks the fix. We test two cheap structural mitigations. A verdict-from-rationale judge decides from the question and rationale only, which cuts the direct edge. A counterfactually paired rationale judge writes its critique on both $X(a)$ and $X(a')$ and must reconcile them, which targets the indirect edge. The prediction is falsifiable: the first should reduce bias in proportion to the NDE share, and the second in proportion to the NIE share. Both are checked for accuracy cost on JudgeBench.

Baselines

Naive reference point: the unmitigated judge. We report its total verdict shift under cue swaps, computed exactly as in The Silent Judge on the same 100 ELI5 and 100 LitBench pairs. Reproducing their numbers is a sanity check. For GPT-4o, they report a recency verdict shift of +30% on ELI5 and +16% on LitBench, an EXPERT-versus-UNKNOWN shift of +18% on ELI5, and 0% cue acknowledgment everywhere.

Strongest existing audit: the verdict-shift plus cue-acknowledgment protocol. This is the Silent Judge audit, extended with an LLM-based detector of cue mentions to give it the fairest chance. It is the strongest competitor on the diagnostic question because it is what practitioners already run. We beat it if our decomposition makes a correct prediction it cannot make. Specifically, the audit treats CAR = 0 as "rationale clean", while a large NIE predicts that a separate model reading only the rationale will reproduce the bias. We test that prediction directly.

CALM robustness rate. The robustness rate from Justice or Prejudice? is computed on identical perturbations. No reported per-model numbers were found on Lacuna for the CALM suite, so we recompute it ourselves.

Strongest mitigation: counterfactual verdict averaging. The judge is queried on both $X(a)$ and $X(a')$ and the verdicts are averaged. This generalizes the swap augmentation used in JudgeLM: Fine-tuned Large Language Models are Scalable Judges, which reduced position delta bias from 13.11 to 1.63. Averaging enforces invariance to the named attribute by construction, so it is the hardest baseline on that attribute. We match it on number of judge calls per item.

For accuracy, the reference numbers come from JudgeBench. GPT-4o scores 50.86% with the vanilla prompt and 56.57% with the Arena-Hard prompt, Claude-3.5-Sonnet scores 64.29%, and o3-mini (high reasoning effort) scores 80.86%. A mitigation "beats" averaging if it matches its bias reduction on the target attribute at no worse JudgeBench accuracy, or if it reduces bias on held-out attributes that averaging was not told about.

Key ablation: same-cue transplant. This isolates the counterfactual ingredient. Any "effect" it shows is noise, which sets the threshold for calling an NDE or NIE real.

Why not just…?

Why not just strip the metadata? For explicit labels like dates and author tags, stripping works, and we say so plainly. That case is our clean testbed, not our target. For attributes carried by the text itself, deletion fails. Counterfactual Fairness shows that "unaware" predictors still shift under counterfactual interventions through proxies. Keyword ablation in Flexible text generation produced fluent text only 46.6% of the time, versus 95.7% for LLM rewrites. Stage 3's entangled-attribute arm measures how much bias survives stripping.

Why not just average verdicts over counterfactual rewrites? This is the real threat as a mitigation, and we expect it to be very hard to beat on any attribute named in advance. Our claim is not that it fails there. Our claim is that it is silent on where bias lives. Averaging costs a factor of two per attribute and does nothing for attributes nobody thought to rewrite. If Stage 4 shows verdict-from-rationale judging matches averaging on held-out attributes in direct-dominant judges, the decomposition earns its place. If not, we will report the decomposition as a diagnostic for audit validity only.

Why not just read the rationales with a better cue detector? The Silent Judge already found 0% acknowledgment, and a better detector of explicit mentions cannot see a rationale that is merely harsher. The NIE measures exactly that hidden channel. The rationale-only reader test shows whether reading rationales could ever be a sufficient audit.

Why not just train robustness in with preference optimization, as BiasScope does? BiasScope reports that its bias-augmented DPO improves robustness. It also reports that GPT-4o still errs 74.7% of the time on JudgeBench-Pro. JudgeBench finds preference-fine-tuned judges such as JudgeLM and PandaLM worse than random on style-neutralized reasoning. We include a DPO-trained judge and decompose it before and after training. This shows whether training removes bias or merely relocates it from the rationale to the direct path, which would make its rationales less, not more, trustworthy.

Proposed Experiments

The judges are GPT-4o and Claude-3.5-Sonnet through rationale-as-prior-message, open-weight Llama-3.1 and Qwen2 judges with full logit access, and JudgeLM as a fine-tuned judge. The data are ELI5 and LitBench pairs with Silent Judge cues, CALM-style verbosity and authority perturbations on GSM8K and TruthfulQA, CivilComments-Identities items with LLM-generated identity rewrites, and JudgeBench and JudgeBench-Pro for accuracy and trap robustness.

The decisive checks are four.

First, the estimator recovers all three planted mechanisms in Stage 2, including the cancellation case, beyond the same-cue noise floor.

Second, on real judges, NDE, NIE, and their interaction account for the total verdict shift.

Third, wherever the NIE is large despite 0% cue acknowledgment, a rationale-only reader reproduces the bias. That result would show that "clean" rationales carry disguised bias.

Fourth, the bias reduction from verdict-from-rationale versus paired-rationale judging tracks the measured NDE versus NIE shares across judges and attributes, at JudgeBench accuracy within a few points of the unmitigated judge. If the decomposition fails the third and fourth checks, it is a curiosity rather than a tool, and we will say so.