Language models are now routinely graded by other language models. A "judge" model reads a candidate answer and decides whether it is correct, and that decision becomes a benchmark score or a reinforcement-learning reward. A separate body of work shows that language models solving problems are easily thrown off by irrelevant but on-topic text. One fluent sentence about a sibling who also bought apples can wreck a math solution. This note asks what happens when those two facts meet. If a judge is as distractible as a solver, then a wrong answer padded with plausible, topic-matched filler could be graded as correct. Because the filler reads naturally, the filters that catch today's known judge attacks would not see it. A policy trained against such a judge could also discover this padding by itself, with no attacker involved.
We propose to measure this channel and to test one specific defense, which we call extract-then-isolate judging. Before the judge sees anything, an extractive selector copies out only the spans of the candidate that form its derivation from question to final answer. A step verifier then checks each extracted step in a fresh context that holds only the question and the preceding steps. The point of the design is that distraction stops being a hidden influence on the judge's attention. A distractor is either dropped, or it is carried into the chain as an explicit step that can be checked and rejected. The open question is whether this beats the simplest strong competitor, which is adversarially fine-tuning the judge on attacked examples. We set up the experiments so that this competitor can win.
Core Claim
We make two claims. The first is empirical: reference-free correctness judges show a contextual distraction vulnerability under in-topic, entity-overlapping padding. Optimization against such judges finds this vulnerability, and perplexity filters cannot detect it. The second is a design claim. Removing the shared context between distractor text and the reasoning being checked should cut judge false positives under unseen distractor families more than adversarial training does, while losing less recall than "be suspicious" prompting. Adversarial training only covers the attack families it was trained on, and suspicious prompting makes judges reject good answers.
The empirical claim fills a gap between two literatures. Judge-gaming work studies three kinds of attack. AdvJudge-Zero uses short control tokens such as | Step |, which flipped verdicts on wrong math solutions. Raina et al. use greedy-searched adjective strings. Ramprasad et al. use content-free assertions that the summary is faithful. None of these papers uses padding that is semantically coherent and on-topic. Distraction work, on the other hand, studies solvers rather than graders. Shi et al. found that in-topic sentences with role-name overlap are the most damaging. Wang et al. used tree-searched coherent distractions and reported roughly 45% average degradation. Both papers motivate our first claim: they show the vulnerability exists in solvers, and we ask whether it transfers to graders.
Raina et al. complicate the picture. They report that pairwise comparison is much more robust than absolute scoring. Our first claim predicts this robustness will not hold against coherent distractors. We test that prediction directly instead of assuming it. The Silent Judge found verdict shifts of up to +30% from short metadata cues, with zero acknowledgment of those cues in the judge's own reasoning. This suggests distraction-driven verdicts will also go unexplained, which is why we measure the damage behaviorally rather than by reading the judge's rationale.
Research Program
Setting. The work targets reference-free correctness judging, where the judge is not given a gold answer. When a gold answer with an extractable final value exists, rule-based matching already solves grading, and we use it only to compute ground truth. Reference-free judging is the regime that matters for process grading, for domains without verifiers, and for rubric rewards. The main testbed is GSM8K, MATH and AIME solutions, following AdvJudge-Zero's setting. Transfer is tested on HealthBench rubric grading as used by Mahmoud et al. and on SciAux explanations from Zhao et al..
Attack construction. We adapt the Error-Guided Perturbation tree search from Wang et al. In their work the search maximizes solver error. Here it maximizes the judge's probability of saying "correct" on a solution that is known to be wrong. The search is constrained to insert only fluent sentences that share the problem's entities and do not change the stated final answer. We build three distractor families: entity-overlap narrative in the style of GSM-IC, plausible-but-irrelevant intermediate computations, and self-verification prose such as "checking: 12 × 3 = 36, consistent." Attacks are searched on one open judge family and transferred to others. We also run a policy-discovery experiment. We train Qwen2.5-7B-Instruct with GRPO, the policy-optimization method Mahmoud et al. used, against a reference-free judge on MATH. We then measure whether the frequency of on-topic padding rises together with the judge's false-positive rate. Mahmoud et al. observed rubric exploitation climbing from 39% to 65% during training, so drift of this kind is a realistic expectation.
Defense. The extractive selector may only copy contiguous spans of the candidate, and we enforce this with exact span matching. This addresses the obvious failure mode, a rewriter that quietly repairs wrong solutions. Selection follows the logic of Superposition Prompting, which scores retrieved documents on separate paths and prunes irrelevant ones before generation. We borrow that score-and-prune step and apply it to the candidate text rather than to retrieved documents. The step verifier then judges each extracted step with only the question and the previous extracted steps in context. The final verdict is the conjunction of the step verdicts.
The key argument rests on how distraction fails. Shi et al. show that distracted solvers pull extraneous numbers into their calculations. If the selector does the same, the distractor's number appears as an explicit step, the arithmetic becomes checkable, and the step fails. That failure is a false negative on a solution that was already wrong, which is the safe direction. The unsafe direction is that the selector drops a genuinely needed step from a correct solution. That is the robustness tax, and we measure it separately.
Baselines
The naive reference point is the vanilla judge: the same judge models scoring the full candidate under a standard prompt. We use both binary verdicts and 1–5 scores. Everything else is compared against its false-positive rate on attacked wrong solutions and its true-positive rate on clean correct ones.
The strongest existing method is adversarial fine-tuning of the judge. AdvJudge-Zero reports that LoRA adversarial training cut Omni-Judge's false-positive rate on AIME from 96.46% to 1.80% against its control-token attack. Wang et al. similarly report that preference training on distracted versus clean pairs restored solver robustness. This is the competitor most likely to beat us. We give it the same judge backbone and the same attacked training pool we use to tune our selector, plus a matched compute and hyperparameter budget. We then evaluate both methods on a held-out distractor family and on padding discovered by the GRPO policy. Beating it means a lower false-positive rate on the held-out families at a true-positive rate within two points of it on clean data. There is no reported result on Lacuna for any judge under in-topic semantic distractors, so the first experiment establishes that number.
Two further competitors are evaluated. The first is instructed prompting, which follows Shi et al. Their "feel free to ignore irrelevant information" instruction raised code-davinci-002 micro-accuracy from 72.4% to 77.8%. We also include the stronger "manipulation-aware" judge prompt from Gaming the Judge. The second is pairwise comparative judging, following Raina et al.
Two ablations isolate the key ingredient. The first verifies steps in isolation but does not extract, so each sentence of the raw candidate is judged locally. The second extracts but judges the extracted chain in a single context. If either ablation matches the full method, the claim that both extraction and isolation are needed is wrong. A thinking-mode version of every judge is also run, because Zhao et al. found that extended reasoning made Qwen3-8B worse under misleading context (77.03% with thinking on versus a 96.22% clean baseline).
Why not just…?
Why not just tell the judge to ignore irrelevant text? The evidence from solvers is weak. Shi et al. found macro accuracy stayed below 30% across noise types even with instructions. Wang et al. found instructions ineffective against tree-searched distractors. Khalifa et al. found that suspicion prompts reduce false positives at the cost of rejecting genuinely successful agents. We still run this baseline in full, because judges might differ from solvers.
Why not just adversarially fine-tune the judge? This is the real threat. The AdvJudge-Zero result above shows that it works very well against the attack family it was trained on. Our bet is that it generalizes poorly across distractor families, because coherent padding has no fixed surface form to learn. The held-out-family and policy-discovered-padding evaluations settle the question. If the fine-tuned judge holds up on both, we will report that the extraction defense is unnecessary in this regime.
Why not just filter by perplexity or penalize length? Perplexity flagged Raina et al.'s strings with F1 between 0.7 and 0.8. Our distractors are fluent by construction, and we will report the filter's detection F1 on them to show the gap directly. A length penalty is cheap and partly effective. However, it also penalizes legitimately long derivations, and an attacker can pad within the length budget. We include a length-matched control, in which distractors replace an equal number of tokens from the solution, to separate distraction from plain verbosity.
Proposed Experiments
The first experiment tests whether vulnerability exists at all. Judge false-positive rates on wrong GSM8K, MATH and AIME solutions, with and without each distractor family, should show increases comparable to the solver degradations Wang et al. report. Pairwise judging should collapse as well, which would contradict Raina et al.'s robustness finding for this attack class.
The second experiment tests whether optimization discovers the exploit. During GRPO training, both padding frequency and judge false-positive rate should rise, while held-out rule-based accuracy should stay flat or fall.
The third experiment is the main comparison. We plot false-positive and true-positive curves for the vanilla judge, the instructed judge, the adversarially trained judge, the two ablations and the full method, on seen and held-out families. We then retrain the GRPO policy against the defended judge and check whether padding still increases.
The proposal fails if any of three outcomes occurs. Distractors may barely move judges. Adversarial training may generalize to the held-out families. Or extraction may cost more than a few points of true-positive rate on clean correct solutions. Each outcome is reported as a finding in its own right.
References
- AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens
- Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment
- Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation
- Large Language Models Can Be Easily Distracted by Irrelevant Context
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Reward Hacking in Rubric-Based Reinforcement Learning
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation