When a large language model is used as a judge, meaning it is asked to decide which of two answers is better or whether an agent finished its task, it usually writes a justification before giving its verdict. We propose to rephrase the judge's own instructions several ways and then compare two things across the rephrasings: whether the verdict changes, and whether the stated reasons for it change. The hypothesis is about one specific pattern. When the verdict stays fixed across rephrasings but the reasons offered for it keep changing or contradicting each other, the justification is probably a rationalization written after a decision the model had already made on other grounds. This pattern is the signature of the errors that matter most in automated evaluation: systematic preferences for longer, more polite, or citation-laden answers that do not reflect quality.
Existing defenses miss these errors, and the reason is structural. Averaging or voting over prompt variants cancels random noise. A bias that pushes every variant the same way survives the vote. Instability-based confidence scores flag only cases where the verdict flips, and Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity shows that such detectors track consistency rather than correctness. Reasons, unlike verdicts, have content. A biased judge must invent a different defensible-sounding story each time it is prompted differently, while a judge that has actually found a decisive flaw tends to name that flaw every time. Reason stability, measured conditional on verdict stability, may therefore expose the confidently wrong verdicts that voting cannot touch.
Core Claim
Among judge verdicts that agree across all prompt variants, the dispersion of the decisive reasons cited across those variants predicts verdict errors and causally unfaithful justifications. Verdict-only signals have no information on this subset by construction. We further claim that rebuilding the verdict from the reasons that recur across variants, after each recurring reason is checked against the evidence, corrects a meaningful share of these errors at a fixed call budget.
To make this precise, we run one judge model under $K$ semantically equivalent judge prompts. For each variant $k$, the judge produces a verdict $v_k$ and a justification. A separate extraction step reduces each justification to a set $R_k$ of decisive claims, such as "response B misstates the boiling point" or "response A ignores the second constraint." We cluster these claims by meaning across variants. Let $A$ be the fraction of variants that agree with the majority verdict. Let $S$ be the average pairwise overlap, measured as the size of the shared set of claim clusters divided by the size of the combined set, among the majority-verdict variants. The dissociation score is
$$D = A\,(1 - S),$$
which is high when the verdict is stable but its supporting reasons are not. The verdict-instability signal used by On the Relation between Sensitivity and Accuracy in In-Context Learning is essentially $1 - A$. On items where $A = 1$, that signal is constant and $D$ is the only remaining information.
Grounding in Prior Work
Four strands of prior work motivate the gap. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge injects twelve kinds of non-substantive change into the candidate answers, such as added length, fake citations, or polite tone, and shows that frontier judges are swayed by them. It never varies the judge's own prompt and never asks whether the judge's explanations are honest. Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation shows that judges trust fabricated progress claims in an agent's reasoning, raising false positive rates by 20–30 points, and that majority voting over sampled judge outputs does not fix content fabrication. That paper studies the unfaithfulness of the agent being judged. We study the unfaithfulness of the judge. Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge names "verdict bias," where a judge picks a winner first and justifies it afterwards, and counters it by training a plan-then-execute judge. It uses one plan from one prompt and offers no test of whether verdict bias remains. Articulate Intuition or Genuine Analysis? finds that a judge given nine rubric dimensions collapses them into one "analytical-trace" factor. This constrains our design: rubric juries can share one hidden heuristic, so diversity must be measured in what the reasons say, not assumed from how the prompts are worded.
One paper supplies the closest method and the strongest competitor. Auto-Prompt Ensemble for LLM Judge (APE) learns 16 evaluation dimensions from judge errors, treats them as jurors, and overrides the base verdict when their summed votes clear a threshold. It aggregates verdicts and never inspects whether jurors' reasons cohere. We reuse its jury idea but aggregate at the level of reasons.
Three further papers supply the measurement tools. The interventions in Measuring Faithfulness in Chain-of-Thought Reasoning and FRIT test whether removing or corrupting a reasoning step changes the answer. We use them to validate $D$ against the judge's actual causal dependence on its stated reasons. FaithCoT-Bench (FaithCoT-Bench) shows that LLM judges detect unfaithful reasoning better than logit methods, with F1 between 65 and 77. Its counterfactual methods fail on knowledge-heavy text, which is why we measure unfaithfulness through cross-prompt reason consistency rather than step deletion alone.
One result constrains the corrective step. The finding of On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models that forcing faithfulness can cost up to 20% accuracy warns that the reason-based rebuild could hurt accuracy. We therefore report accuracy and faithfulness jointly. We also note that our method edits no weights, so it is a direct test of whether that trade-off holds without training.
Research Program
The work proceeds in three stages.
The first stage builds the prompt variants and the reason extractor. For each benchmark we write $K = 8$ judge prompts that differ in surface form but not in task. The variants cover paraphrased instructions, persona and tone changes in the style of ProSA (ProSA), order of presented answers, and output-format constraints. Following the two-stage decomposition of VALOR-EVAL, a second model call turns each justification into atomic decisive claims. A third pass clusters claims by meaning. We validate extraction and clustering against human annotation on 300 justifications before trusting $D$.
The second stage tests whether $D$ measures what we say it does. On a subset of items, we take each variant's top decisive claim, delete it or replace it with a contradicting claim in the justification, and re-elicit the verdict, an adaptation of early answering and of FRIT's step substitution to judges. A justification counts as causally unfaithful if the verdict survives removal of its own decisive reason. The prediction is that high-$D$ items are disproportionately unfaithful by this test.
The third stage is the corrective procedure, reason-level ensembling. We pool the claim clusters cited across all variants and keep those that recur in at least a set fraction of variants. Each kept claim is checked against the inputs, meaning the two responses, the reference answer, or the agent's actions and screenshots. The verdict is then recomputed from the surviving claims. This is the analogue, for generated reasoning, of averaging prompt embeddings toward a stable center, which the prompt-ensembling literature leaves open for generative outputs. Here the center is the set of reasons that persist across phrasings, not an average of vectors.
Baselines
The naive reference point is the single default judge prompt with greedy decoding. Its verdict accuracy is what all methods must improve on. Its verbalized confidence provides the trivial error detector. APE reports that verbalized confidence for GPT-4o has an expected calibration error of 0.317 (Auto-Prompt Ensemble for LLM Judge).
The first competitor is self-consistency: sample $K$ judgments from the default prompt at nonzero temperature and take the majority vote, using vote agreement as confidence. It is the cheapest way to spend extra compute. Gaming the Judge reports that it helps against style manipulations but not content fabrication. We run it with exactly the same number of judge calls as our method, plus as many extra samples as our extraction and verification calls cost, so that it is never under-budgeted.
The second competitor is verdict instability across prompt variants, in the style of SENSEL. It uses the same $K$ prompts as our method but only the verdicts, scoring confidence by agreement. On the Relation between Sensitivity and Accuracy in In-Context Learning reports a +4.1 point gain in area under the F1–coverage curve over maximum softmax probability on GPT-J-6B. This baseline is also our central ablation. It shares every prompt and call with our method and differs only in ignoring reasons, so any gain of $D$ over it isolates the contribution of reason content.
The strongest existing method is APE. We re-implement its dimension generation on Skywork preference data as described, and use 16 jurors with its gated override. On RewardBench it reports raising GPT-4o human agreement from 87.2% to 90.5%, OffsetBias accuracy from 81.5% to 91.0%, and expected calibration error to 0.041. These are the numbers to beat with GPT-4o as judge. We also report EvalPlanner's 93.9 RewardBench score with Llama-3.1-70B as a trained-judge ceiling. It is not a like-for-like competitor, since it requires training. We do test whether $D$ applied on top of EvalPlanner-style plan-then-judge prompts still finds errors.
For detecting the judge's unfaithful reasoning, we compare against the Faithful-Judge detector of FaithCoT-Bench, which reports F1 between 65 and 77 on candidate-model reasoning. Applied to judges, it is the strongest available single-pass detector. No reported result was found on Lacuna for any method detecting unfaithfulness in a judge's own reasoning.
Beating these baselines means three things. First, $D$ must exceed verdict instability in error-detection AUROC on the full set and must be meaningfully above chance on the unanimous-verdict subset, where verdict instability is exactly at chance. Second, reason-level ensembling must match or exceed APE's 90.5% RewardBench agreement and 91.0% OffsetBias accuracy at no greater call count. Third, it must reduce the bias-induced flip rate on CALM-style perturbed pairs below APE's.
Why not just…?
Why not just use self-consistency over temperature samples? Sampling from one prompt reproduces that prompt's blind spots. APE documents this as a recognition deficit, and Gaming the Judge finds that voting fails against fabricated content. The deeper issue is that a systematic bias produces the same wrong verdict in every sample, so vote agreement is high exactly where the judge is wrong. Our unanimous-subset analysis tests this directly. If self-consistency agreement carries signal there, our premise is weakened.
Why not just measure verdict flips across prompt variants? This is the real threat. If biased errors turn out to be mostly unstable verdicts, verdict instability captures nearly everything and reason extraction is wasted cost. The answer is empirical and comes from our core ablation. We report what fraction of judge errors on CALM-perturbed and OffsetBias pairs are unanimous across all eight variants. If that fraction is small, we will say verdict instability suffices and narrow the contribution to the faithfulness audit. Rethinking Hallucinations gives reason to expect a large consistent-error share, since its consistency-tracking detectors miss correctness, but we have not seen this measured for judges.
Why not just use APE's rubric jury? APE votes on verdicts. Kahneman4Review's finding that a judge collapses nine rubric dimensions into one factor suggests APE's jurors may share one heuristic and vote together when wrong. We test this by computing $D$ over APE's own 16 jurors' justifications. If APE's errors cluster at high $D$, reason-level aggregation should help it. If they do not, APE's diversity is real and our corrective step adds little, though the audit still applies.
Why not just instruct the judge to ignore length, tone, and citations? The reverse chain-of-thought study (Measuring and Mitigating Post-Hoc Rationalization) finds that suppression instructions increase hidden anchoring. Gaming the Judge finds that manipulation-aware prompting makes judges reject valid work. We include one debiasing-instruction variant among the eight prompts and also run it as a standalone baseline, reporting both its accuracy and its loss of recall.
Proposed Experiments
The main experiment uses RewardBench and its OffsetBias subset with GPT-4o and Llama-3.1-70B as judges. We add CALM-style perturbed versions of RewardBench pairs, adding length, fabricated citations, polite or frustrated tone, or swapped order to the worse response while keeping it worse. This gives a controlled population of bias-induced errors. For agent evaluation, we apply Progress Fabrication rewrites to web-agent trajectories as in Gaming the Judge, using its 800 trajectories if available and reproducing the rewrite procedure otherwise. The measured outcome is the false-positive inflation that voting fails to remove. To check whether $D$ transfers beyond verdict tasks, we have judges label FINE-CoT trajectories as faithful or unfaithful and compute $D$ on the judge's own labels.
There are three decisive checks. The first is the unanimous-subset AUROC of $D$ for predicting judge errors. Values near 0.5 falsify the core claim. The second is the rate at which high-$D$ justifications fail the deletion test, compared with low-$D$ ones. If the rates are equal, $D$ measures reason wording noise rather than rationalization. The third is whether reason-level ensembling lowers the bias-induced flip rate and the Progress Fabrication false-positive inflation relative to APE and self-consistency at a matched call count, while holding or improving clean accuracy. A gain in robustness bought with a clean-accuracy drop would reproduce the accuracy–faithfulness trade-off of Tanneru et al. and would be reported as such.
One confound gets its own control. A correct judge might legitimately cite different true reasons across variants. We therefore separate reason dispersion into two parts: claims that differ but are all verified as true, and claims that are contradictory or fail verification. We expect only the second part to carry the error signal.
References
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- Auto-Prompt Ensemble for LLM Judge
- Measuring Faithfulness in Chain-of-Thought Reasoning
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning
- On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation