Large language models are now the default graders of other models' outputs, and they are biased in ways that careful prompting does not remove. They reward longer and more heavily formatted answers. They are swayed by filler that imitates deliberation, such as "wait, let me think about it". They favor authority language, and their written justifications omit the cue that actually moved them. The root problem is architectural. The same forward pass reads every surface feature of both candidates, reasons about them, and writes the verdict, so any feature of the text can reach the decision. We propose to cut that path. A frozen model first reads each candidate alone and may output only verbatim spans copied from that candidate, one small, fixed-size packet per rubric item. A second pass reads only these anonymized packets and decides. The copying is enforced during decoding by trie-constrained decoding: a prefix tree (trie) of all substrings of the candidate masks every next token that would leave the candidate's actual text. The extractor can select but cannot paraphrase, editorialize or embellish.
The claim is that biases carried inside the candidate text (length, formatting, reflection filler, embedded authority phrases) are sharply reduced when the verdict can see only equal-budget verbatim evidence. A second claim is that the verbatim-only constraint is the ingredient that matters, not the two-stage split itself. A free-text extractor can launder bias into its summaries, which is the rationalization pattern documented in The Silent Judge. Selecting spans is a much narrower channel. What bias remains must pass through which spans get chosen, and we can measure that directly. The whole method runs on frozen models, zero-shot or few-shot, with no preference fine-tuning.
Core Claim
The judgment is factored as
$$V = g\big(q, r, \pi(E_A, E_B)\big), \qquad E_i = f(q, r, c_i) \subseteq \mathrm{Spans}(c_i), \quad |E_i| \le B,$$
where $q$ is the task prompt, $r$ the rubric, $c_i$ candidate $i$, $\mathrm{Spans}(c_i)$ the set of contiguous token spans of $c_i$, $B$ a token budget shared by both candidates, $\pi$ a random ordering with identities hidden, $f$ the constrained extractor and $g$ the verdict model. The extractor never sees the other candidate, any metadata, or any bandwagon or authority text in the prompt. The verdict model never sees raw candidates.
Under this factoring, candidate-external cues cannot reach $g$ at all. In-text cues reach $g$ only if $f$ spends budget quoting them. Length differences are erased beyond $B$. Every statement $g$ makes can be traced to a span ID.
We are explicit about where the contribution lies. Isolating external cues is achievable by simply deleting metadata, and we credit that baseline. The new claims concern in-text biases, and the gap between span-only and paraphrase extraction.
Relation to Prior Work
Three diagnostic papers motivate the gap.
The Silent Judge injects provenance and recency labels. It finds verdict shifts of up to +30% for GPT-4o on ELI5, alongside a 0% rate of the justification acknowledging the cue. Free-text justifications are therefore not a trustworthy audit trail.
Assessing Judging Bias in Large Reasoning Models shows that extended chain-of-thought (the model writing out intermediate reasoning steps before answering) does not protect against bias. DeepSeek-R1 falls from 73% to 37% on Emerton-DPO under bandwagon bias. The paper also documents a mid-reasoning "criteria shift" and a bias it calls superficial reflection, where filler phrasing sways the verdict. Its prompt-only mitigations reach reductions of at most 27%, and are inconsistent on factual tasks.
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena established position and verbosity bias.
Two systems supply the closest methods.
RULERS requires judges to output verbatim quotes and sentence IDs inside a JSON schema. It then applies a post-hoc "evidence gate" that caps scores when the quotes are invalid, and reaches a quadratic weighted kappa (QWK, an agreement measure for ordinal scores) of 0.7276 on ASAP 2.0 against 0.4319 for holistic scoring. RULERS differs from our design in two ways. Its judge still sees both full candidates when it decides, and quote validity is checked after generation rather than enforced during it.
Graph-constrained Reasoning builds a trie over knowledge-graph paths and reports a 100% faithful-path ratio, against 48–62% without the constraint. It then hands these paths to a separate model for free reasoning. We reuse that two-stage structure, with the trie built over candidate text rather than a graph. The official codebase supplies trie decoding.
Other work constrains the design.
Mitigating Bias in Locally Constrained Decoding shows that masking one token at a time can trap generation in dead ends. Our automaton avoids this by construction, because every prefix of a valid span can be closed as a valid quote. The distortion that paper describes, renormalizing probability over the allowed tokens, still applies, so we measure span quality rather than assuming it.
FormalJudge names unconstrained fact extraction as its weak point. Our extractor is a candidate fix for that step, although we do not test it here.
FairJudge and JudgeLM debias judges through fine-tuning. We treat them as trained reference points, not as the method we are trying to replace.
Research Program
We will build three components.
Substring automaton. For each candidate we build an automaton over its own token sequence that accepts any contiguous span. This is size-linear and built once per candidate. A second small trie covers rubric-item IDs and a closed label set: supports, contradicts, absent.
Extractor decoding loop. The loop alternates between the rubric label trie and quote mode inside quote delimiters. It enforces at most $K$ spans and $B$ total tokens per rubric item, with the same budget for every candidate. The extractor may write a short free-text scratchpad before quoting, but that scratchpad is discarded and never forwarded.
Verdict stage. The verdict model receives both packets in random order with neutral labels and may reason freely. We also test a variant in which every claim in its reasoning must cite a span ID, enforced with the same trie machinery.
Factors and analyses. We vary zero-shot versus few-shot extractor prompts, since in-context examples may improve span selection. We vary the budget $B$ and test standard versus reasoning-model verdict stages. Our main analytic tool is selection bias: we inject an in-text cue into a candidate and measure how much the extracted packet changes, both in edit distance and in whether the cue itself gets quoted. This separates bias that survives through selection from bias removed by the bottleneck.
Models. All experiments use open-weight frozen models, 8B to 70B, including one reasoning model. Logit masking requires access to the model's output probabilities, so closed API judges appear only as unconstrained references.
Baselines
Naive reference. A zero-shot chain-of-thought judge that sees both candidates with standard prompting. It is the "do nothing" point against which every bias is measured.
Cue stripping plus pointwise scoring. We delete all metadata and prompt-level social cues, then score each candidate separately. This is the strongest cheap competitor on external cues, and we expect it to match our method there by construction. Beating it means lower bias on in-text cues at equal human agreement.
Prompt mitigations. We run the system-prompt, in-context and self-reflection mitigations from Assessing Judging Bias in Large Reasoning Models, which report bias reductions of up to 19%, 27% and 16% respectively. They use the same models and the same number of in-context examples as our few-shot extractor.
Swap averaging. We judge both orderings and average, which doubles cost.
RULERS-style judging, re-implemented on the same frozen models. This is the strongest existing inference-time method. The judge sees both candidates, emits schema-constrained quotes, and passes through a post-hoc evidence gate with up to three retries. We match its total token budget to ours. Its reported ASAP 2.0 QWK of 0.7276 with GPT-4o-mini is the number our method must match on that benchmark. On our other benchmarks it is re-run rather than quoted.
Trained reference. FairJudge-8B, which reports 65.52% consistency on FairJudge-Benchmark-1K against 50–55% for GPT-4-class judges. It shows what fine-tuning buys; it is not a requirement to beat.
Ablations. Three ablations isolate the key ingredients.
Paraphrase extractor. This is the same pipeline with an unconstrained extractor holding the same budget. It tests whether the verbatim constraint matters.
Post-hoc-validated extractor. The extractor generates freely, invalid quotes are rejected, and it retries. This tests whether enforcing the constraint during decoding matters, as opposed to checking it afterwards.
Joint-context extractor. The extractor sees both candidates and any cues. This tests whether isolation matters.
Why not just…?
Why not just strip the metadata? For provenance, recency, bandwagon and authority cues placed outside the candidates, stripping probably works as well as our method. We will report it as doing so. It cannot remove length, formatting, reflection filler, or authority phrasing written inside an answer, and those are the cases that decide whether this proposal holds up.
Why not just prompt the judge to be impartial, possibly with examples? The reported gains are partial and unstable on factual items: at most 27% on preference data and 16% on factual data in Assessing Judging Bias in Large Reasoning Models. The 0% cue-acknowledgment finding in The Silent Judge also suggests the model does not know what to ignore. We run these prompts on identical models and budgets, so the comparison is direct.
Why not just check quotes after generation, as RULERS does? This is a real threat to the decoding mask, as opposed to the bottleneck. With enough retries, post-hoc validation may produce quote validity equivalent to the mask. The post-hoc-validated extractor ablation settles this. If it matches the mask on bias and agreement at similar cost, we will conclude that the bottleneck carries the effect and the mask is an efficiency detail. The mask would then retain only the advantage of zero retries and guaranteed validity.
Why not just fine-tune a debiased judge? Fine-tuning works, as FairJudge and JudgeLM show. It needs curated data, however (JudgeLM's 105,000 seed judgments), and it does not apply to frozen deployments. We include FairJudge as a reference and ask whether a frozen model with the bottleneck closes most of that gap.
Proposed Experiments
In-text bias suite. We start from human-labeled pairs in MT-Bench human judgments, the PandaLM benchmark (GPT-4 agreement 66.47%) and the JudgeLM validation set. From these we build controlled counterfactuals in which the worse answer receives a padded but content-neutral elaboration, heavier formatting, inserted reflection filler, or an embedded authority sentence such as "As a domain expert, …". The metric is the verdict shift rate between the original and perturbed pairs, the same measure used by The Silent Judge.
No reported baseline magnitude for verbosity-shift rate was found on Lacuna, so all methods are measured fresh. The decisive check has two parts. First, the span-only pipeline must cut verdict shift clearly below the RULERS-style and cue-stripping baselines. Second, the paraphrase ablation must show larger shifts, and inspection should show its summaries inflating the perturbed answer.
External-cue suite. We reuse the provenance and recency injections on ELI5 and LitBench, where GPT-4o shows recency verdict shifts of +30% and +16%. We add the bandwagon, authority and position perturbations on Emerton-DPO and the MMLU-Pro subsets, where R1 drops from 73% to 37% under bandwagon. We expect our method to tie cue stripping here. The point of this suite is to confirm that isolation works, not to claim a new effect.
Accuracy cost. A bottleneck that removes bias by removing information is no gain. We measure agreement with humans on MT-Bench (where GPT-4 reaches roughly 80–85%) and PandaLM, QWK on ASAP 2.0 against RULERS' 0.7276, and cross-mode consistency on FairJudge-Benchmark-1K. We predict a real cost on holistic creative judgments (LitBench), where quality lives in whole-text flow that fragments cannot capture. We will report the budget $B$ at which agreement recovers, trading bias against accuracy along that curve.
Selection-bias analysis. For every in-text perturbation we record how often the extractor quotes the injected cue under each rubric item, both zero-shot and few-shot. This tells us whether the remaining bias lives in span selection, and whether examples in the prompt suppress it.
Decision rule. The proposal succeeds if two conditions hold together. The span-only pipeline must lower in-text verdict shift relative to both RULERS-style judging and the paraphrase ablation, without dropping human agreement by more than a few points on MT-Bench and PandaLM. It fails as a distinct idea if the paraphrase or post-hoc-validated extractors perform equally well. In that case the finding would be that isolation plus a fixed budget is sufficient, and the trie is unnecessary.
References
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Assessing Judging Bias in Large Reasoning Models: An Empirical Study
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation
- Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
- Mitigating Bias in Locally Constrained Decoding via Tractable Proposals
- FormalJudge: A Neuro-Symbolic Paradigm for Agentic Oversight
- FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges