Language models are now routinely used as judges: given a question and two candidate answers, a strong model picks the better one and explains why. These verdicts rank models on leaderboards and supply the preference labels used to fine-tune the next generation of models. Judges are not neutral, though. Changing an author tag from "UNKNOWN" to "EXPERT", or a date from 1950 to 2025, flips a large share of GPT-4o's verdicts. Its written justifications never mention the tag; they invent quality-based reasons instead. Every known shortcut of this kind was found by a person who guessed it in advance and wrote a template to test it. A judge debiased against that hand-written list is only as robust as the guesses behind it.
We propose to search for a judge's shortcuts automatically and then train the judge to ignore what the search finds. The search uses gradient-guided discrete sampling, a family of methods that treat token sequences as samples from an energy function and use one backward pass to estimate which token swaps would most change that energy. Here the energy rewards short text, inserted into a region of the input that cannot change which answer is correct, that flips the judge's verdict. Because the sampler carries a Metropolis–Hastings correction (an accept/reject step that makes it draw from the target distribution instead of collapsing onto one optimum), it returns a varied population of verdict-flipping cues rather than one adversarial string. We then turn the discovered flips into preference pairs and fine-tune the judge with Direct Preference Optimization (DPO), a method that trains a model directly on "this output is better than that one" pairs without a separate reward model. The claim to test is this: a judge debiased on sampled cues generalizes to shortcut families nobody enumerated, and a judge debiased on hand-written templates does not.
Core Claim and Positioning
The work closest to this proposal is diagnostic or template-bound. The Silent Judge holds content fixed, manipulates provenance and recency tags, and measures the Verdict Shift Rate (the fraction of verdicts that change) and the Cue Acknowledgment Rate (how often the rationale mentions the cue). It reports a recency shift of +30% for GPT-4o on ELI5 and an acknowledgment rate of 0%, but it proposes no fix and searches only cues the authors wrote down. RM-Bench does the same for style, applying fixed concise, detailed, and markdown transforms. It finds that leading reward models fall to 46.6% average accuracy when the wrong answer is better formatted, and that scale does not help: Nemotron-340B-Reward scores 69.5% against 70.1% for an 8B model. These two papers motivate the proposal. They show the bias is real and survives both chain-of-thought prompting and scale. Their probe sets also serve as our held-out evaluation.
On the mitigation side, Learning LLM-as-a-Judge for Preference Alignment (Con-J) trains a Qwen2-7B judge with DPO on contrastive correct and incorrect judgments and tests robustness only against synthetic lengthening. Response-conditioned modeling (Rc-DPO) removes length bias by building pairs that hold the response fixed and vary the prompt constraint. We reuse both training recipes. What we change is where the bias pairs come from: a sampler instead of a fixed transform. Neither paper addresses provenance, recency, or unknown cues.
The search machinery comes from Oops I Took A Gradient, whose Gibbs-with-Gradients sampler uses a first-order estimate of every single-token energy change, and from Principled Gradient-Based MCMC for Conditional Sampling of Text. The second paper's p-NCG and GwL samplers add exact discrete accept/reject steps for language. It also shows that continuous-relaxation-then-round methods are asymptotically biased. There, these samplers were steered by static sentiment and topic classifiers. We point them at a judge's verdict.
That paper also constrains the design. It finds that ℓ2 embedding geometry gives erratic proposals while ℓ1 behaves well, so we adopt ℓ1. The Gibbs-with-Gradients paper warns that its first-order estimate degrades on non-smooth energies. A judge's verdict probability may well be such an energy, which is why the accept/reject correction is not optional here.
The Search Objective
Take a prompt $x$ and an answer pair $(y_a, y_b)$ whose ground-truth winner is $y_a$. Let $z$ be a sequence of $L$ tokens placed in a designated non-substantive slot: an attribution or metadata header, a preamble before an answer, or a formatting-only wrapper. These slots are chosen so that, by construction, no string in them changes which answer is correct. Any verdict change caused by $z$ therefore counts as bias, with no need for a human to judge semantic equivalence. We sample from
$$p(z) \propto \exp\big(\log P_J(b \mid x, y_a \oplus z, y_b) + \alpha \log p_{\text{LM}}(z) - \lambda \lVert z \rVert_{\text{cue}}\big),$$
where $P_J(b \mid \cdot)$ is the judge's probability of preferring the worse answer $b$, and $y_a \oplus z$ means $z$ is inserted into the slot attached to answer $a$. The term $p_{\text{LM}}$ is a small reference language model's likelihood, which keeps cues fluent so they resemble text that could plausibly appear in real data. The penalty $\lVert z \rVert_{\text{cue}}$ counts non-padding tokens, so short cues are preferred. Inverse temperature is folded into $\alpha$ and $\lambda$. We run p-NCG multi-token proposals interleaved with GwL single-token moves, using one gradient of this energy per step.
Running many chains across many answer pairs produces a cue population. We cluster it by embedding similarity and also check it manually for families: authority claims, dates, confidence hedges, hidden formatting, and anything new. The existing templates would cover only some of these.
Research Program
The work has three parts. First, we build the probing sampler for white-box judges where gradients are available. These are a Qwen2-7B-Instruct judge trained with the Con-J recipe on Skywork-Reward-Preference-80K, and Skywork-Reward-Llama-3.1-8B as a scalar-score judge. We measure which cue families appear, how often each flips the verdict, and whether the rationale ever mentions it. We also test whether the cues transfer, unchanged, to closed judges such as GPT-4o, scored by black-box shift rate. Transfer would make the method useful for auditing judges we cannot open.
Second, we turn the cues into debiasing data. For each flip, the chosen judgment is the clean-input verdict and rationale, which agree with ground truth. The rejected judgment is the judge's own flipped verdict and rationale under the perturbed input. Training uses the Con-J objective, DPO plus a supervised term. Because the cue pool is a sample rather than a fixed list, we refresh it from the updated judge every round, in an adversarial loop.
Third, we test generalization along the axis that matters: from discovered cues to cue families the training never saw.
Baselines
The naive reference point is the Con-J-style judge trained on Skywork preferences with no bias augmentation, alongside the untuned Qwen2-7B-Instruct judge. Con-J reports 81.0% on Infinity-Preference against 75.0% for GPT-4o, so this baseline is strong on accuracy. For open 7B judges, the file contains no reported Verdict Shift Rate on the ELI5 and LitBench probes ("no reported result found on Lacuna" for open-judge shift rates), so we measure it ourselves. The closed-model reference is GPT-4o's +30% recency shift and +18% EXPERT-versus-UNKNOWN shift on ELI5 from The Silent Judge.
The strongest competitor is template counterfactual augmentation. It applies the same Con-J DPO training to pairs built from hand-written perturbations: Silent Judge provenance and recency tags, RM-Bench style transforms, and the Rc-DPO length-conditioned pairs. This is the obvious way to debias a judge, and on the families it covers it should do well. Both arms get the same base model, the same number of augmented pairs, the same judge-forward-pass budget for data construction, and the same hyperparameter sweep over DPO β and learning rate. Beating it means a lower shift rate on held-out cue families at equal or better clean accuracy. Clean accuracy here means RewardBench overall (the ArmoRM-Llama3-8B reference score is 89.0) and RM-Bench average and Hard accuracy (reference: 70.1% average and 46.6% Hard).
Two search-method baselines run at matched compute. The first, a black-box LLM rewriter, prompts a strong model to invent metadata or preambles meant to sway a judge. The second, random token search, samples slot contents from the reference language model and keeps flips. Two ablations isolate our ingredients. One removes the accept/reject step and greedily takes the best gradient-estimated swap, which tests whether a calibrated sample rather than a single optimum is what yields coverage. The other replaces discrete sampling with continuous relaxation and rounding in the style that the conditional-sampling paper found unfaithful.
Why not just…?
Why not just augment with the templates we already know about? This is the real threat. If biases come in a handful of families and the Silent Judge and RM-Bench templates already cover them, templated augmentation will match our method at lower cost. The held-out-family experiment below settles this directly, and we will report a null result if templates generalize as well. Our prior that they will not rests on RM-Bench's finding that models fail even on its own style dimensions, and on the 0% acknowledgment rate, which suggests judges absorb cues the judges themselves never surface. The open question is whether sampled cues are more varied than templated ones in a way that matters for generalization.
Why not just ask a strong LLM to brainstorm biasing cues? This is cheap, needs no gradients, and also works on closed judges. Its weakness is that it proposes what a language model expects to bias a judge, which roughly matches what humans already guessed. It does not propose what actually moves this judge's verdict probability. It is a matched-budget baseline, and we compare cue-family coverage and held-out generalization against it.
Why not just use chain-of-thought rubrics or a bigger judge? The evidence in hand already argues against both. The Silent Judge shows that step-by-step justifications coexist with +30% recency shifts and 0% acknowledgment, and RM-Bench shows that a 340B reward model is no more robust than an 8B one.
Why not just do best-of-N random search over perturbations? Random search over a vocabulary of size $K$ with $L$ slot positions needs on the order of $L \times K$ judge calls to explore single-token neighbors. Gradient proposals estimate all of them in one backward pass; the Gibbs-with-Gradients paper estimates a roughly 870× speedup on categorical models. Our matched-compute random-search baseline checks whether that advantage holds on judge energies, which may be rough.
Decisive Experiments
The central test is a leave-family-out split. The sampler is run, and its cues are used for training, while all date-like and authority-like tokens are excluded from the slot vocabulary, so that provenance and recency cues cannot be discovered. We then evaluate on the Silent Judge ELI5 and LitBench provenance and recency probes and on RM-Bench Hard. The templated baseline is trained without those same families, with style and length templates only. The proposal succeeds if the sampled-cue judge shows a clearly lower shift rate on the excluded families than the template judge, with RewardBench and RewardBench 2 scores no worse than the no-augmentation judge. RewardBench 2's random baseline is 25%. The reverse split also runs: training on templates only and evaluating against a fresh sampler run on the debiased judge, which measures how many new flipping cues remain.
Secondary checks cover four things. One is the coverage ablation: whether the corrected sampler produces more distinct cue clusters per judge call than the greedy and relaxed variants. Another is cross-judge transfer of discovered cues to GPT-4o. A third is whether training on flipped rationales raises the Cue Acknowledgment Rate on held-out probes or merely suppresses the flip. The last is a sanity check that the sampler's flip rate on the debiased judge falls across adversarial rounds without the judge collapsing toward a fixed position or a single answer.
If templates match sampled cues on held-out families, the conclusion is that judge biases are few and enumerable. That is worth knowing, and it would redirect effort toward cataloguing them rather than searching.
References
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
- Learning LLM-as-a-Judge for Preference Alignment
- Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling
- Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
- Principled Gradient-Based MCMC for Conditional Sampling of Text