Which Distance Should Pick a Judge's Examples?

When a large language model is used as a judge, for example to decide which of two answers better follows an instruction, a few worked examples placed in the prompt can change its verdicts a great deal. This is in-context learning: the frozen model adapts to a task from examples alone, with no weight updates. In practice those examples are either fixed or retrieved by cosine similarity between sentence embeddings of the new case and a pool of labeled cases. That retrieval rule measures similarity of topic. A judge, however, fails for reasons of structure: a response that looks polished but quietly violates a constraint, a provenance tag reading "EXPERT," a recent date. Two cases about the same topic can sit on opposite sides of the judge's failure mode. Two cases about different topics can share exactly the trap that fools it.

This proposal argues that for LLM judges, the distance used to retrieve examples matters more than the rule used to choose among the retrieved candidates. It tests a specific replacement. Distance is measured in the judge's own hidden-state space, at the position where it emits its verdict, after reweighting toward the directions that predict when the judge's zero-shot verdict is wrong. The idea comes from an old result outside NLP. When clustering movement trajectories, Morris and Trivedi found that swapping the similarity function moved the correct-clustering rate from about 0.57 to above 0.99, while swapping the clustering algorithm barely mattered. We ask whether the same ordering of importance holds for choosing a judge's examples, and whether an error-aware distance closes the gap on adversarial judging benchmarks.

Core Claim

The claim has two parts. First, across representations, distances, and selection rules, the choice of representation and distance accounts for more of the variation in judge accuracy than the selection rule does. This holds on benchmarks built around superficial traps. Second, one particular distance retrieves examples that share the judge's likely mistake, and this lowers both adversarial error and shortcut-driven verdict shifts. That distance is a Mahalanobis-style distance (one that stretches some directions and shrinks others) over verdict-position hidden states, whitened toward error-predictive directions. The examples it retrieves also transfer to closed models whose internals we cannot read.

Concretely, let $h_x$ be the hidden state of an open frozen judge at its verdict token for case $x$, taken from an intermediate layer. Let $U$ be a matrix whose columns span the top discriminant directions separating cases the zero-shot judge gets right from cases it gets wrong. These directions are fit on a labeled pool. The retrieval distance between a query $q$ and a pool candidate $c$ is

$$d(q,c)=\sqrt{(h_q-h_c)^\top\left(UU^\top+\lambda I\right)(h_q-h_c)},$$

where $\lambda$ controls how much ordinary Euclidean geometry is kept. This distance is not arbitrary. Easier to Judge than to Find reports that successful and failed query–context pairs are linearly separable in an LLM's hidden states, with probe AUROC above 0.90. So a direction encoding "this judgment is about to go wrong" plausibly exists. Retrieving neighbors along it should surface examples that contain the same trap, with the correct verdict attached.

Relation to Prior Work

Three demonstration-selection methods motivate the proposal, and all three keep cosine similarity over sentence embeddings as the retrieval step. Revisiting Demonstration Selection Strategies retrieves a cosine top-k pool and reranks it by the model's conditional entropy. It reports 77.9% versus 76.1% average accuracy on classification tasks with Llama-2-7B, but cases dropped by the cosine step are never seen by the reranker. MarginSel selects examples near the model's zero-shot decision boundary, gaining 2–7 F1 points on ambiguous classification. Its boundary is defined by predicted label sets, not by a geometry. DiSP, from Easier to Judge than to Find, uses small classifiers to judge whether a demonstration set will work. None of these three work on evaluation tasks, and none changes the distance. We keep their selection rules as factors in our design and vary the geometry underneath them.

On the judging side, Calibrating LLM-Based Evaluator induces written grading criteria from 8–12 human-labeled examples, which are held fixed for every query. We instead select examples per query by geometry, and we compare against and combine with its rubric. Evaluating LLMs at Evaluating Instruction Following and The Silent Judge supply the failure modes we target: superficial-quality traps, and unacknowledged provenance and recency shortcuts. These two papers also act as constraints. Chain-of-thought can rationalize wrong verdicts, and judges never admit to using the cue (a 0% cue acknowledgment rate). A fix that relies on the judge reasoning its way out is therefore unlikely to work, which is why we change which examples it sees rather than asking it to explain more. Fine-tuned Judge Models are Task-specific Classifiers constrains the alternative of training a judge. In-Context Learning Creates Task Vectors supplies an analysis tool, the idea that examples act through a compact internal state, which we use to check the mechanism. Its results cover only single-token tasks. Whether judgment behaves the same way is part of what we test.

Research Program

We build a pool of labeled pairwise judgments with known correct verdicts. Each pool item is a triple: instruction, response A, response B. The pool is drawn from the expert-voted MT-Bench and Chatbot Arena judgments in Judging LLM-as-a-judge, plus held-out portions of LLMBar. For each item we record the open judge's hidden states at the verdict position under both answer orders, and whether its zero-shot verdict is correct. The discriminant directions $U$ are fit on the pool only, never on test items.

The core study is a factorial grid. The first factor is the representation: sentence embedding of the instruction, sentence embedding of the full triple, or the judge's verdict-position hidden state. The second is the distance: cosine, Euclidean, plain Mahalanobis, or the error-whitened distance above. The third is the selection rule: nearest neighbors, MarginSel-style boundary selection, or conditional-entropy reranking. Every cell supplies the same number of examples to the same judge. We then decompose accuracy variance across factors. This directly tests whether the trajectory-clustering ordering, where distance dominates algorithm, reappears.

Two mechanism checks follow. In the first, we measure whether the examples chosen by error-whitened retrieval move the query's verdict-position state across the probe's error boundary. If gains arrive without this movement, the hidden-state story is wrong. In the second, we extract a task-vector-style state from the example block, following Geva et al., and patch it into a query-only pass. This tests whether evaluative judgment compresses into a steerable vector the way single-token tasks do.

Baselines

The do-nothing reference is the zero-shot judge with the standard pairwise prompt and order swapping. For the strongest closed model, Evaluating LLMs at Evaluating Instruction Following reports about 73% accuracy for GPT-4 on the LLMBar Adversarial set, and near chance (about 50%) for LLaMA-2-70B. The same paper's Rules+Metrics+Reference prompting reaches 82.8% with GPT-4. That is the strongest prompting competitor on this benchmark and the number to beat. On LLMBar Natural, GPT-4 reaches 93.5% per Huang et al., leaving little headroom, so Natural serves as a no-harm check. For shortcut bias, The Silent Judge reports a recency verdict shift rate of +30% for GPT-4o on ELI5 and +16% on LitBench. A method beats this baseline if it substantially reduces these shifts without lowering accuracy on cue-free pairs.

The standard retrieval competitor is cosine top-k over sentence embeddings, which is kNN in-context learning. There is no reported result on Lacuna for kNN-retrieved examples on LLMBar or MT-Bench judging, so we establish it ourselves. The strongest selection-rule competitors are conditional-entropy reranking (Peng et al.) and MarginSel (Ambati et al.), both run on cosine candidates. Their reported gains (77.9 vs. 76.1 average accuracy, and 2–7 F1) come from classification, not judging. For pointwise scoring, the competitor is AutoCalibrate's fixed calibrated rubric, which reports +10.4% Spearman correlation over vanilla GPT-4 on NewsRoom (Liu et al.).

Fairness rules apply to every method. All methods see the same pool, the same number of examples, the same judge, and the same prompt template. Pool-derived quantities, namely $U$, the reranker entropy, and MarginSel's zero-shot labels, are computed under the same budget of judge forward passes. Hyperparameters such as $\lambda$, layer choice, and $k$ are tuned on a validation split with equal grid sizes for every method.

The key ablations isolate the ingredient. Setting $U$ to zero leaves plain Euclidean distance in the judge's space, which separates the effect of the representation from the effect of error-whitening. Fitting $U$ on randomly relabeled correctness tells us whether the error signal itself matters. Success means the whitened distance beats both ablations and every cosine-based cell by a margin larger than seed-to-seed variance.

Why not just…?

Why not just use cosine top-k, which is cheap and standard? Cosine on sentence embeddings rewards topical overlap. MarginSel's results show that topically similar examples tend to be easy, prototypical ones that do not teach a fine boundary. Whether this matters for judging is unmeasured, so cosine top-k is a full cell of the grid rather than an assumption. If it ties the whitened distance on LLMBar Adversarial, the core claim fails, and we will report that.

Why not just use an uncertainty-based selection rule like MarginSel or entropy reranking? These rules change which candidates win but inherit the candidate set from cosine retrieval. MarginSel's authors also note that it degrades when the base model's zero-shot distribution is poorly calibrated. Judges on adversarial pairs are often confidently wrong, which is exactly where uncertainty is uninformative. The factorial grid settles this directly. If selection rule explains more variance than distance, the proposal's ordering claim is refuted.

Why not just train an error probe and flip the verdicts it flags? This is the most serious threat. For binary pairwise judgments, a probe that detects likely errors could correct them without any examples, and we run probe-flip as a baseline with the same probe used to fit $U$. It cannot direct a pointwise score, so it has nothing to say on SummEval or NewsRoom. It also cannot help a closed judge like GPT-4o, whose hidden states are unavailable. Our transfer setting covers that case: examples retrieved with an open model's geometry are given to the closed judge. If probe-flip matches our method on LLMBar for open judges, we will narrow the claim to pointwise scoring and black-box transfer.

Why not just fine-tune a judge or a learned retriever? Huang et al. show that fine-tuned 7–13B judges reach only 53–70% on LLMBar Natural, against GPT-4's 93.5%, and collapse to about 24% under a change of judging scheme. This is consistent with training producing narrow classifiers. Learned retrievers such as SeDPO are slow according to Easier to Judge than to Find. Our method fits only a low-rank direction matrix on a frozen model's states, and it keeps the general judge intact.

Why not just write a better rubric or add chain-of-thought? Rules+Metrics+Reference prompting is a baseline, and we also stack our retrieved examples on top of it to test whether the gains add. The evidence from LLMBar and The Silent Judge, that chain-of-thought rationalizes traps and judges never acknowledge cues, suggests that written instructions alone will not remove the shortcut shifts.

Proposed Experiments

The judges are Llama-3.1-70B and Qwen2.5 (open, used for hidden states) and GPT-4o (closed, used for the transfer test). The primary benchmark is LLMBar Adversarial, using leave-one-subset-out splits so that the manually built traps and the GPT-4-crafted distractors are each tested with pool examples from the other source. That split checks that retrieval is not simply matching the generator of the trap. LLMBar Natural and MT-Bench expert agreement serve as no-harm checks.

For shortcut bias, we reproduce The Silent Judge's cue injections on ELI5 and LitBench and report the verdict shift rate. The pool for this experiment contains cue-bearing pairs in which the cue points to the wrong answer. Our prediction is that cue directions load on $U$, so error-whitened retrieval will pull these counterexamples for cue-bearing queries. We verify this by counting retrieved counterexamples per method.

For pointwise scoring, we run SummEval and NewsRoom with Spearman correlation as the metric. We compare against AutoCalibrate's fixed rubric, and against that rubric combined with per-query examples.

The decisive check is the variance decomposition on LLMBar Adversarial. The proposal succeeds if the representation and distance factors together explain more variance than the selection rule. It also needs the error-whitened distance to beat cosine top-k, the Euclidean and shuffled-label ablations, and probe-flip, and to exceed the 82.8% prompting reference when combined with it. The proposal fails if cosine top-k or probe-flip matches it.