When a large language model (LLM) is used as a judge, it is shown a user prompt and two candidate responses and asked which one is better. It often returns a confident verdict for reasons unrelated to the prompt: one answer is longer, better formatted, labeled "expert," or placed last. This proposal borrows a check from word translation. There, a candidate translation is kept only if translating it back recovers the original word, and that test removes "hub" words that look like good matches for everything. The analogous test for a judge is simple. If a judge says response A beats response B for prompt x, the verdict should depend on x. Show the same pair under a closely related but different prompt, or ask the judge which of several prompts makes A the better answer. If the verdict does not move, or the judge cannot pick out x, then the preference was never about the question. We flag or down-weight that verdict.
This matters because pairwise comparison is now the default way to evaluate open-ended generation and to build preference data for alignment. The known defenses are narrow. Swapping the order of A and B only tests whether presentation order affects the verdict. Length regression only removes length. Neither catches a judge that prefers the same response no matter what was asked. We measure success against human pairwise judgments, which are the reference that comparative evaluation trusts most.
Core Claim
A pairwise verdict carries two components. One is prompt-grounded: A is better at this task. The other is prompt-independent: A is better-looking in general, the evaluation analogue of a hub in nearest-neighbor retrieval. We claim three things. First, the prompt-independent component can be estimated by re-judging the same pair under counterfactual prompts. Second, verdicts dominated by it disagree with humans far more often than verdicts that pass the check. Third, a reverse-prompt filter gives a better trade-off between human agreement and the fraction of verdicts kept than order swapping or length control, and it works with judges that return only text.
Write $p(x)$ for the judge's probability that A beats B under prompt $x$. For text-only judges, we estimate it by averaging verdicts over both presentation orders and a few samples. Let $N(x)$ be a set of $K$ distractor prompts retrieved as nearest neighbors of $x$ in an embedding space. The grounded margin is
$$g(x) = p(x) - \frac{1}{K}\sum_{x' \in N(x)} p(x'),$$
which is the preference that exists only because of the true prompt. This is the subtraction that hubness corrections perform in lexicon induction. The retrieval version of the check works in the reverse direction. The judge sees the ordered pair "A is better than B" together with the shuffled prompt set $\{x\} \cup N(x)$ and names the prompt for which the preference holds most strongly. The verdict passes if the judge recovers $x$. Following the fusion of forward and backward scores in Enhancing Bilingual Lexicon Induction via Bi-directional Translation Pair Retrieving, the final score combines the forward verdict with the reciprocal rank of $x$, weighted by a single parameter $\lambda$ tuned on held-out human labels.
Where This Sits
The mechanism comes from reverse-verification methods on discrete tasks. The lexicon-induction work above re-ranks translation candidates by how highly the source word ranks when the retrieval is run in reverse, which breaks hubs. Self-Augmented In-Context Learning for Unsupervised Word Translation keeps an LLM translation only if back-translation returns the exact source word. SPE: Symmetrical Prompt Enhancement for Fact Probing checks a predicted object by predicting the subject back, and gains most on the hard split, where frequent-token bias is worst. A Multi-turn Machine Reading Comprehension Framework with Rethink Mechanism for Emotion-Cause Pair Extraction adds a reverse question that prunes false emotion-cause pairs. All four operate on discrete outputs where "recovering $X$" means string identity. We carry the idea to pairwise judgments, where the recovered object is the prompt itself. This also answers the limitation noted in Self-Consistency Preference Optimization, whose majority voting needs an extractable final answer and so cannot reach open-ended tasks.
The motivation comes from evidence that judges use prompt-independent shortcuts. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena reports GPT-4 pairwise agreement with human experts of about 80–85%. It also documents position and verbosity bias and introduces the order-swap-then-tie rule. Judging the Judges shows that order inconsistency is systematic and concentrated where the quality gap between candidates is small. The Silent Judge shows that injected recency and provenance cues shift verdicts, by up to +30% for GPT-4o on ELI5, while judges never mention the cue. The cues in that study are, by construction, independent of the prompt, which is exactly what our check targets.
The closest existing work falls into three groups, and each reverses something different. Order swapping (Judging LLM-as-a-judge, Judging the Judges) reverses the positions of the two candidates. EvolvR reverses scores to attack its own reasoning chains during synthetic data generation. Reverse Preference Optimization rewrites verifiable instruction constraints so the chosen response complies. None of them reverses the conditioning to ask whether the prompt can be recovered from the preference. None applies to subjective quality without rules or ground-truth scores.
Two papers constrain the design. Who can we trust? LLM-as-a-jury for Comparative Assessment re-weights judges by how consistent their comparisons are across a graph. It needs logits, and its authors warn that bias shared across judges survives aggregation. Our check is meant to detect shared prompt-independent bias, but it must be compared against that method. Ties Matter shows ties can exceed half of pairs in modern MT evaluation. A filter that simply converts hard cases to ties can inflate agreement, so we report agreement at matched coverage and use tie-calibrated accuracy.
Research Program
We will build the check in three variants. The subtraction variant computes $g(x)$ from $K$ re-judgments. The retrieval variant is a single listwise call asking the judge to identify the prompt, with the list order shuffled across repeats to control position effects. The fused variant combines both. We will study how distractor difficulty matters: random prompts from the benchmark, prompts from the same category, and nearest-neighbor paraphrase-adjacent prompts. Distractors that are too easy make the check vacuous. Distractors that are too close make the correct response legitimately good for both prompts. This is the evaluation version of the asymmetric-mapping failure of bi-directional filters noted in the research file.
A central risk is that some prompt-independent preference is legitimate, since humans also reward fluency. To measure the human reference for $g$, we will run a small comparative human study. Annotators judge about 500 response pairs drawn from MT-Bench and ELI5, each under its true prompt and under one nearest-neighbor distractor, with three annotators per item. If humans keep their preference under distractors at a similar rate to judges, the check removes signal humans actually use. If humans' preferences move and the judge's do not, the gap measures judge shortcutting directly. This study decides whether the filter should target $g(x)$ near zero or match the human distribution of $g$.
Baselines
The naive reference is the raw single-order verdict of the same judge, with every verdict kept.
The standard practical baseline is order swapping from Judging LLM-as-a-judge: evaluate (A, B) and (B, A) and treat disagreement as a tie or as a dropped verdict. We treat it as the main competitor because it is what practitioners run today and because it also costs extra judge calls. The reported point of comparison is GPT-4's roughly 80–85% agreement with human experts on MT-Bench. Our filter uses more calls than swapping, so we compare at equal numbers of judge calls. Swapping gets extra samples to reach the same budget. We also compare at equal coverage, meaning the same fraction of verdicts kept. Beating it means higher human agreement at every coverage level between 50% and 90%, and not only at one operating point.
For length, the baseline is the post-hoc regression of Length-Controlled AlpacaEval, which raised Spearman correlation with Chatbot Arena from 0.94 to 0.98. We apply it at the level of individual pairs by thresholding the length-adjusted preference. It gets the same held-out tuning set as our $\lambda$. For judge aggregation, the baseline is BT-σ from Who can we trust?, which reports 50.50% SummEval Spearman correlation against 45.15% for probability averaging. We run it only on open-weight judges where logits exist, with the same judges and pairs.
Two ablations isolate the key ingredient. The first replaces nearest-neighbor distractors with random prompts, which tests whether hard counterfactuals matter. The second uses subtraction alone versus retrieval alone. A further control re-judges the pair under the true prompt $K$ extra times, with no distractors. It matches compute, so it tests whether any gain comes from simply sampling more.
Why not just…?
Why not just swap the order and call disagreements ties? Swapping detects only order effects. A judge that always prefers the response labeled "EXPERT", or the longer one, is perfectly consistent under swapping, and The Silent Judge shows exactly such cues moving verdicts. Swapping also discards the small-gap pairs that Judging the Judges identifies as where inconsistency concentrates. The cue-injection experiment below tests this directly. Swapping should flag almost none of the cue-induced flips, and the reverse-prompt check should flag most of them.
Why not just control for length? This is a real threat. If the prompt-independent component is mostly length, Length-Controlled AlpacaEval does the job with a single regression. On the other side, Rethinking LLM-based Preference Evaluation argues that length is entangled with genuine information content, so penalizing it can punish good answers. We will settle this by regressing $g(x)$ on the length difference. If length explains most of its variance and length control matches our human agreement at equal coverage, we will report the reverse check as unnecessary for length. Its remaining claim would then rest on non-length cues and on pairs of similar length.
Why not just sample the judge more times and take the majority? Repeated forward sampling concentrates on whatever shortcut the judge already favors. SPE reports the analogous frequency bias in fact probing, which only the reverse query breaks. The compute-matched control that re-judges under the true prompt tests this. If it matches our filter, the idea fails.
Why not just use a judge-reliability model like BT-σ? It needs logits and weights judges rather than individual verdicts. Its authors also note that bias shared across judges persists. We run it where possible and report whether combining it with our per-verdict check helps.
Proposed Experiments
The main testbed is the MT-Bench human expert pairwise judgments released with Judging LLM-as-a-judge. Its 80 prompts are few, so distractors are drawn from the other MT-Bench questions and from the Chatbot Arena prompt pool. We add ELI5 and LitBench pairs from The Silent Judge for long-form and creative text. Judges include GPT-4o and open-weight models of several sizes. The primary metric is agreement with human pairwise labels as a function of coverage, reported as the area under that curve and scored with tie-calibrated accuracy following Ties Matter.
The first decisive check is cue injection. We apply the provenance and recency manipulations of The Silent Judge and count what fraction of the resulting verdict flips each filter catches. Because the injected cue does not depend on the prompt, it should survive a change of prompt, and our check should detect it. A result showing the check catches cue-induced flips at a rate no better than order swapping would falsify the core claim.
The second decisive check is the human counterfactual study described above. It compares how much human preferences and judge preferences move under distractor prompts.
The third check is downstream use. We filter UltraFeedback preference pairs with our check before direct preference optimization, a method that trains a model directly on chosen-versus-rejected pairs. We compare against the Best-of-$N^2$ pairing of DCRM, which reports a 0.59 Pearson correlation with AlpacaEval gains, and against unfiltered pairs, holding the number of training pairs fixed. Evaluation uses length-controlled AlpacaEval and MT-Bench. As a reference point, What Makes Good Data for Alignment? reports 7.55 on MT-Bench for its Mistral-7B model after preference optimization. If the filter improves evaluation agreement but not downstream training, we will report that plainly. The claim would then be limited to evaluation.
References
- Enhancing Bilingual Lexicon Induction via Bi-directional Translation Pair Retrieving
- Self-Augmented In-Context Learning for Unsupervised Word Translation
- SPE: Symmetrical Prompt Enhancement for Fact Probing
- A Multi-turn Machine Reading Comprehension Framework with Rethink Mechanism for Emotion-Cause Pair Extraction
- Self-Consistency Preference Optimization
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
- Judging the Judges: A Systematic Investigation of Position Bias in Pairwise Comparative Assessments by LLMs
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- EvolvR: Self-Evolving Pairwise Reasoning for Story Evaluation to Enhance Generation
- Reverse Preference Optimization for Complex Instruction Following
- Who can we trust? LLM-as-a-jury for Comparative Assessment
- Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration
- Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
- Rethinking LLM-based Preference Evaluation
- DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning