When Reasoning Judges Overthink: Candidate-Side Distraction on Contamination-Free Problems

Language models are now routinely graded by other language models. The newest graders are "reasoning judges": models that write out a long private chain of thought before issuing a verdict. They are trusted because that deliberation is supposed to catch subtle errors. Work on solvers points the other way. When a reasoning model reads misleading or irrelevant text, its step-by-step thinking tends to build a justification for the misleading material instead of discarding it, and its output grows longer as its accuracy falls. We ask whether the same thing happens to judges. When the answer being graded is wrapped in plausible but irrelevant or misleading prose, does the judge talk itself into accepting a wrong answer? And does the extra length of its own thinking give that away?

Two existing results make this hard to measure. First, the standard benchmarks for testing judges are contaminated: the problems appear in pretraining data, so a judge can recognize the right answer from memory and never engage with the candidate's argument. Second, most judge benchmarks have no verifiable ground truth. We address both by building judge tests from procedurally generated problems whose correct answers are known and cannot have been memorized. We then inject controlled noise into the candidate responses while holding their correctness fixed. On top of this we test a cheap warning signal that needs no labels: how much longer the judge thinks when it reads a candidate than when it reads the bare problem.

Core Claim

Our central hypothesis has three parts. First, reasoning judges are much more vulnerable to noise inside the candidate response than current judge benchmarks show. Second, the gap is largest on uncontaminated problems, because on contaminated ones a memorized answer anchors the judge. Third, the judge's errors are preceded by measurable inflation of its own reasoning trace. If all three hold, current practice overstates judge reliability in exactly the regime where judges matter most: novel problems without a reference answer. It would also mean a per-item length ratio can flag unreliable verdicts at almost no cost.

The direction that motivates this most directly is the "rationalization trap". Thinking in a Crowd shows that Qwen3-8B with thinking enabled falls from 96.22% to 77.03% on its SciAux questions when two misleading snippets are added, while its token counts spike. That finding concerns solvers answering questions. We move the misleading text into the object being judged.

Work on reasoning judges provides the opposing claim that we put to a test. Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases reports DeepSeek-R1 at 91.18% on RewardBench and documents residual length and formatting bias. JudgeLRM trains judges with reinforcement learning that rewards correct verdicts and reports gains over DeepSeek-R1. Neither paper measures how much the judge thinks, or tests judges on candidates that carry distractors.

Two further papers supply the mechanism we expect. The First Impression Problem finds that overthinking arises when a model's immediate gut answer conflicts with its step-by-step derivation, rather than from difficulty alone. BiasTrace finds that overthinking behaviours in reasoning traces predict socially biased outputs with ROC-AUC 0.98. It establishes that trace behaviour can predict bad decisions, but only for demographic bias in solvers, and it uses a second LLM to annotate the traces. Our signal needs no annotator. It is a token count compared against a per-item reference, and it applies to technical judging errors.

Research Program

The test bed starts from DyVal, which generates arithmetic, logic and graph-reachability problems from random directed acyclic graphs (graphs of dependent steps with no cycles) at four controllable depths, D1 to D4. Models that score well on static benchmarks score near zero on these problems, which is evidence that memorization does not transfer.

For each problem we collect candidate solutions from several solver models, verify them against the generated answer, and form matched pairs. The same candidate's core derivation and final answer appear either clean or wrapped in one of four kinds of noise. The first is irrelevant but on-topic text. The second is misleading auxiliary claims in the style of SciAux (resource). The third is locally coherent but wrong justification appended to a correct or incorrect answer. We add it because Local Coherence or Global Validity? shows that GPT-4o tags reasoning errors with only 57.8% accuracy on fluent but globally invalid traces. The fourth is adaptively searched distractors. For these we adapt the error-guided tree search of Adaptive Distraction, which retries perturbations in the direction that caused errors. We point it at the judge's verdict rather than a solver's answer. That paper reports that static noise causes under a 5% drop, while its search causes drops above 45% on solvers, so the adaptive condition is where we expect the real stress.

The contamination contrast needs matched difficulty. We therefore also build the same four noise conditions on GSM8K items, which are plausibly memorized, and on GSM8K variants whose numbers are regenerated so the memorized answer no longer holds. We then compare judge behaviour against DyVal problems binned to the same solver accuracy. The prediction from the first-impression account is specific. On memorized items the judge's gut answer is right, so noise produces little conflict and little inflation. On regenerated and DyVal items the gut answer is weak, so noise should drive both inflation and error.

The warning signal is simple. Let $L(q,c)$ be the number of thinking tokens the judge produces when grading candidate $c$ on problem $q$, and $L(q)$ the number it produces when asked to solve $q$ alone. The excess ratio is

$$r(q,c) = \frac{L(q,c)}{L(q)}.$$

The denominator removes the difficulty of the problem itself, so what remains is the thinking attributable to the candidate. We will test whether high $r$ predicts wrong verdicts, and whether a judge that abstains or re-queries above a threshold on $r$ recovers accuracy.

The judges we will study are Qwen3-8B with thinking on and off, JudgeLRM-7B and 14B, QwQ-32B and DeepSeek-R1. The on/off comparison directly tests whether deliberation helps or hurts a judge that faces distractors.

Baselines

Naive reference. The simplest reference is the same judge with thinking disabled and a plain grading prompt, on the identical matched pairs. This tells us whether deliberation is net positive under candidate noise. On SciAux, thinking-off Qwen3-8B scored 74.87% against 77.03% with thinking on under misleading context, a small margin, so we do not assume the answer. Beating this reference means the thinking judge has a lower false-acceptance rate on noisy wrong candidates.

Structured rubric prompting. The strongest existing mitigation for superficial judge bias is PlanJudge, from Reasoning Model Is Superior LLM-Judge. It makes the judge write evaluation criteria before grading and reportedly lifts DeepSeek-R1 on BiasBench from 65.00 to 97.50. It is the most likely method to neutralize distractors, because a rubric written before reading the candidate could anchor the judge. We run it with the same judges, the same pairs and the same token cap. We count our analysis as adding value only if residual PlanJudge errors under adaptive noise remain above the clean-pair rate, and if $r$ still predicts those residual errors. No reported result found on Lacuna for PlanJudge on candidate-side distraction or on DyVal-derived judging.

Calibrated multi-sample aggregation. The strongest method that spends extra compute is Bradley-Terry-Davidson aggregation over several judge samples, from Distribution-Calibrated Inference Time Compute. It fits a model of pairwise preferences that includes ties, and needs only about 50 to 100 calibration examples. We give it the same total token budget as our excess-ratio gating and the same calibration set. Our signal must reach at least equal verdict accuracy at lower cost, or catch systematic errors that aggregation reinforces, to count as a win.

Uncertainty-based detectors. The detector baselines are raw trace length, the judge's stated confidence, and disagreement across five samples. Each is compared to $r$ by area under the ROC curve for predicting wrong verdicts.

Ablations of the key ingredient. The key ingredient is the per-item denominator $L(q)$. We ablate it by replacing $L(q)$ with a problem-level average across candidates. We also ablate the uncontaminated setting by running the whole analysis on GSM8K alone.

Why not just…?

Why not just tell the judge to ignore irrelevant material? Adaptive Distraction reports that "ignore irrelevant info" instructions gave almost no defence for solvers. The Silent Judge shows judges follow shortcuts while writing justifications that never mention them, so an instruction to be objective may change the explanation without changing the verdict. We include this prompt as a condition rather than assuming it fails.

Why not just have the judge solve the problem first and compare? This is a real threat, and it is close to our denominator run. On shallow DyVal problems a solve-first judge may largely ignore the candidate's noise. It cannot help where the judge cannot solve the problem itself, which is the reason to use a judge on hard or reference-free items in the first place. We run solve-first as a baseline stratified by depth. If it matches our results at D3 and D4, the practical recommendation becomes "solve first," and our contribution shrinks to the measurement study. We will report that outcome if it occurs.

Why not just cap the judge's thinking budget? If inflation is only a symptom, truncating the trace with a stopping-token rule like ThinkBrake might cut cost without touching accuracy, or might cut off the reasoning that catches the error. The First Impression Problem reports that early-exit probes and fine-tuning on short chains hide overthinking without removing its cause. We test fixed caps at matched budgets. If capping fixes the errors, inflation causes them. If it does not, inflation is a diagnostic rather than a cause.

Why not just paraphrase away the style? Preference Leakage finds that paraphrasing removes stylistic leakage, which reaches a leakage score of 37.1% on Arena-Hard. Our misleading and coherent-but-wrong distractors are content, not style, so we expect paraphrase to preserve them. We test this by paraphrasing the noisy candidates with an independent model before judging.

Decisive Checks

The first check is whether false acceptance of wrong candidates under adaptive noise is significantly higher on DyVal and regenerated-GSM8K items than on original GSM8K items at matched solver accuracy. If it is not, the contamination claim fails.

The second check is whether the excess ratio $r$ predicts wrong verdicts better than raw length, stated confidence and sample disagreement, within each depth level. If it does not, the detector claim fails.

The third check is transfer. We fit the $r$ threshold on DyVal and apply it to RewardBench and Arena-Hard pairs, where we inject the same noise types. We measure whether gating recovers accuracy toward the clean-pair level. For reference, DeepSeek-R1 clean RewardBench accuracy is reported at 91.18%. No reported result found on Lacuna for any judge on noise-injected RewardBench.

The fourth check tests whether deliberation itself is the vulnerability, and uses the thinking on/off contrast. We look for thinking-on judges losing more accuracy from clean to noisy conditions than thinking-off judges, while their $r$ rises.