Reachability-Masked Judging: Keeping Shortcut Cues Out of LLM Evaluators' Verdicts

Language models are now routinely used as judges that pick the better of two answers or grade an answer against a rubric. They are also easily swayed by things that have nothing to do with quality. A label saying a response came from an "expert," or that it was written in 2025 rather than 1950, flips GPT-4o's verdicts substantially. Worse, the judge never mentions the label in its written reasoning; it invents content-based praise instead. Because the rationale hides the real cause, reading the judge's explanation cannot tell you whether it was fooled. We propose a different kind of defense. Instead of asking the judge to ignore irrelevant material, we build the judgment as a small graph. Each rubric criterion connects only to the parts of the response that a separate, isolated check found relevant to it. The final verdict is computed only from response parts that can be reached from the criteria, so anything unconnected is never shown to the verdict step.

The idea borrows from a line of work on generating sequences that must respect a graph. There, the model is blocked from producing transitions that the graph does not contain, rather than being trained to avoid them. We apply the same move to information flow inside an evaluator. A cue can influence the verdict only if it first earns an explicit, logged edge to a criterion. This turns the "silent" bias into one that can be inspected. When a cue does get through, the graph records which response unit carried it. The note argues that this masking, and not merely extracting evidence first, is what reduces verdict shifts. It also lays out experiments that would show whether that is true.

Core Claim

The claim has two parts. First, verdict shifts caused by task-irrelevant cues depend heavily on the judge seeing the cue next to the competing candidate inside one comparative prompt. Second, these shifts drop sharply when two conditions hold together. Relevance must be decided one response unit at a time, with no comparison and no verdict at stake, and the verdict step must be shown only the units that passed that check. The second half is what makes the proposal testable. An extract-then-judge pipeline that shows the verdict step everything should not get the same reduction. If it does, the topology plays no role and the simpler method wins.

We also claim a diagnostic benefit. Written rationales showed a 0% Cue Acknowledgment Rate in The Silent Judge. Because the verdict depends only on reachable units, any verdict flip that survives masking must trace to a specific admitted unit. We call the share of surviving flips that trace to a cue-bearing unit the structural acknowledgment rate. It is computed from the graph, not taken from the judge's self-report.

How the Pieces Fit

Each direction plays a different role here.

Topology-constrained generation supplies the mechanism. In path generation, Continuous Trajectory Generation Based on Two-Stage GAN masks next-step transitions using road connectivity. CausalTAD shows that restricting predictions to adjacent nodes removes the pull of transitions that are frequent overall but unreachable locally. That is close to our setting, where a cue such as "EXPERT" is broadly persuasive but unrelated to any criterion. Learning the Markov Order of Paths in Graphs gives the general argument that removing impossible transitions in advance saves the model from having to learn to ignore them. We reuse the hard mask, not any of these models.

Contextual noise sensitivity supplies the failure model and the constraint on design. Breaking Focus and Adaptive Distraction find two things. Frontier models now shrug off fixed-template noise, with drops under 5%. Distractors found by tree search, however, cut accuracy by 45% on average. These papers also find that "ignore irrelevant text" prompts fail. That rules out a prompt-only defense and tells us our evaluation must include an attacker that adapts to our pipeline.

The rationalization effect motivates isolating the decision step. Thinking in a Crowd shows that extended reasoning turns misleading context into longer justifications. Qwen3-8B fell from 96.22% clean accuracy to 77.03% with thinking on. We expect the same in a judge that reasons with the cue in view. Hiding unreachable units from the verdict step is the direct response.

The judge and judge-bias literatures supply the setting and the measurements. The Silent Judge supplies the Verdict Shift Rate (how often a verdict flips when only the cue is swapped), the Cue Acknowledgment Rate, and the ELI5 and LitBench protocols. It tests only plain pairwise prompting and names structural mitigation as open.

The closest structured judges evaluate along a given graph but do not use it to block cues. AgentEval judges each agent step against only its upstream context and raises failure-detection recall from 0.41 to 0.89. However, its graph is the agent's own execution graph, and it assumes the GPT-4o judge is unbiased. PRISM-Physics scores derivations along a causal graph but removes the LLM judge entirely in favor of symbolic matching, which does not carry over to open-ended text. Our graph is not given in advance. It is built per item, with criteria on one side and response units on the other. Its job is to filter what reaches the verdict, not to locate the root cause of an error.

Research Program

The judge has three stages, each a separate model call.

Stage 1: canonicalization. The response is split into atomic units, roughly one claim or one stylistic move each. Each unit is then rewritten so that statements about who wrote it, when, or how credible it is become separate "attribution" units. For example, "As a domain expert, I can confirm X" becomes the claim "X" plus the attribution "author claims expertise." Without this step, a cue embedded in a sentence would ride into the verdict along with the content it is attached to.

Stage 2: edge proposal. For each rubric criterion and each unit, a call that sees only that pair decides whether the unit bears on the criterion and records a one-line reason. This call never sees the other candidate, the other units, or any verdict. Edges therefore cost about (criteria × units) short calls. We will report that cost and test batching units per criterion as a cheaper variant.

Stage 3: verdict. The verdict call sees the criteria, the reachable units of both candidates, and nothing else. Header metadata and unconnected attribution units are never shown to it.

Scope note. We deliberately treat "which criterion legitimately depends on attribution" as part of the rubric. A rubric could, for instance, value citing sources. The mask then admits exactly what the rubric licenses, and that licensing is visible to anyone reading the rubric.

What we will build and analyze. We will build the pipeline for one closed judge (GPT-4o, matching Silent Judge) and one open judge (Qwen3-8B, with thinking on and off). We will build a cue-injection suite in three tiers:

  • Header labels: the original Silent Judge provenance and recency labels.
  • Embedded phrases: the same provenance and recency information written as phrases inside the response text, where rule-based removal is unreliable.
  • Adaptive distractors: cues found by running the tree-search procedure of Adaptive Distraction against each judge configuration, including ours, so the attacker sees the defended system.

For every configuration we measure:

  • Verdict Shift Rate, per cue type.
  • Structural acknowledgment rate, which is undefined for baselines that keep no graph.
  • Clean agreement, meaning agreement with the dataset's human preference signal on uncued pairs. This guards against a judge that becomes robust by becoming uninformative.
  • Tokens per judgment.

Baselines

Plain judge (the naive reference). The plain single-prompt pairwise judge from The Silent Judge, run with the same models and datasets. The paper reports a GPT-4o recency Verdict Shift Rate of +30% on ELI5 and +16% on LitBench, and an EXPERT-versus-UNKNOWN provenance rate of +18%, all with 0% cue acknowledgment. These are the numbers to reduce. No reported result was found on Lacuna for GPT-4o's clean agreement with human preferences on these two datasets, so we will measure it ourselves.

Hardened prompting. The plain judge plus position swapping, with contradictory verdicts discarded, plus an explicit "disregard authorship and date" instruction. This is the cheapest standard practice. Breaking Focus predicts the instruction will not help against embedded or adaptive cues. If the instruction does fully neutralize them, the case for the more elaborate pipeline weakens.

Ensembling. Auto-Prompt Ensemble for LLM Judge, matched to our pipeline's token budget. It represents the view that bias averages out across prompts. We expect it to fail when the cue pushes every prompt in the same direction, as provenance cues do.

Redaction. A model-based scrubber that deletes provenance and recency content before a plain judgment. On header labels this baseline should essentially tie with ours, and we say so up front. Header labels are the sanity check, not the claim. Beating redaction means a lower Verdict Shift Rate on the embedded and adaptive tiers at equal or better clean agreement.

Unmasked extraction (the key ablation). Our Stage 1 and Stage 2 run unchanged, but the verdict call sees all units, with the proposed edges shown only as annotations. This resembles extract-then-generate pipelines such as Ext2Gen. It isolates the hard mask. A second ablation removes canonicalization to measure how much of any gain comes from splitting attributions off from claims.

Robustness training. A distraction-trained judge: Qwen3-8B fine-tuned with Direct Preference Optimization (training on pairs of preferred and rejected outputs) over clean versus cued judgment trajectories. This follows Breaking Focus, which reports roughly 50% robustness gains for smaller models. It is the strongest learned competitor. We give it the same cue generator for training, held-out cue phrasings for testing, and a tuning budget comparable to the compute our pipeline spends on extra inference.

Fairness and the bar for winning. For every baseline we report the same four metrics on the same items, with matched sampling temperature and three seeds. "Beating" a baseline means a lower Verdict Shift Rate on the embedded and adaptive tiers, with clean agreement no more than two points below the plain judge.

Why not just…?

Why not just strip the metadata before judging? For header labels, that is the right answer, and our experiments will show it matching us there. The problem is that real manipulation, and the adaptive search in Adaptive Distraction, does not stay in headers. A scrubber has to anticipate every way of signaling authority or freshness. Our edge-proposal step instead requires every unit to justify its relevance to a criterion on its own, which does not depend on knowing the cue type in advance. The embedded and adaptive tiers settle the question. If redaction matches us there, the masking adds nothing.

Why not just extract the relevant evidence first and judge on that? This is the most serious threat, because our pipeline is an extraction pipeline with one added constraint. The unmasked extraction ablation exists to answer it. Thinking in a Crowd suggests that a reasoning judge which can still see a cue will build justifications around it, so we predict a meaningful Verdict Shift Rate gap between the masked and unmasked versions. If the gap is within noise, we will report that ordinary decomposition, not topology, does the work.

Why not just train the judge to be robust? Preference training on distracted trajectories helped smaller models in Breaking Focus, so it may well be competitive on cue types seen in training. Its weaknesses are that it does not transfer to closed judges, may not generalize to new cue phrasings, and still produces rationales that cannot be audited. The held-out-phrasing test and the closed-model results will show whether these weaknesses matter in practice. The two approaches can also be combined, and we will test the combination.

Why not just ensemble or swap positions? Both reduce variance, but provenance and recency cues produce consistent bias in one direction. Averaging several judges that all lean the same way leaves the lean intact. The token-matched ensemble comparison will show this directly.

Proposed Experiments and Decisive Checks

Main experiment. We run all configurations on ELI5 and LitBench under the three cue tiers.

Check 1: the mask matters. The first decisive check is the gap between the masked pipeline and the unmasked extraction ablation on embedded and adaptive cues. A clear gap supports the core claim; no gap falsifies it.

Check 2: accuracy is preserved. Clean agreement must hold. A defense that cuts the Verdict Shift Rate by making the judge indifferent to content fails.

Check 3: the attacker adapts to the defense. We rerun the tree-search attacker against our pipeline itself. The question is whether it simply moves its effort to winning an edge in Stage 2. If so, the structural acknowledgment rate should reveal it: the surviving flips would trace to admitted cue units, which is still more auditable than a 0% Cue Acknowledgment Rate. We will also test whether giving Stage 2 a stricter licensing rule closes that gap.

Check 4: transfer to step-level grading. We apply the same masking to step-level grading of math reasoning traces. Local Coherence or Global Validity? reports that a GPT-4o judge reaches only 57.8% accuracy on intermediate error tags and is swayed by locally coherent style. We inject misleading auxiliary premises into the traces and ask whether masking helps or hurts error recall. A drop below that 57.8% reference would show that the mask hides context a judge genuinely needs to assess a step. That would limit the method to preference-style judging.