When a large language model is used as a judge, it is asked a question of the form "how good is response $j$ as an answer to instruction $i$?" Part of the answer the judge gives depends on the pair: does this response actually address this instruction? Another part depends only on the response: is it long, fluent, confident, well formatted, or labeled as written by an expert? This proposal asks what happens if we show the judge the same response next to other, similar instructions it was not written for. A response that still scores well there is being rewarded for something other than fitting the task. That is the signature of a "hub," the term from high-dimensional retrieval for a point that looks close to everything. The proposed fix is borrowed directly from word translation. We subtract from each judge score the average score that response earns on its nearest near-miss instructions, and the average score the instruction gives to its nearest near-miss responses. This is cross-domain similarity local scaling (CSLS), applied to the judge's own score matrix rather than to embedding vectors.
The motivation is that LLM judges carry biases that are hard to list in advance. Length-controlled AlpacaEval removes one named confounder by regression. The Silent Judge study shows that injected provenance and recency cues flip verdicts by up to 30% while judges never mention the cue. A correction that removes any preference that does not depend on the instruction, without naming it, would be a more general debiasing tool. It needs no training and no access to the judge's internals beyond a score.
Core Claim
Let $S_{ij}$ be the judge's continuous score for response $j$ given instruction $i$. We use the probability-weighted expected score from G-Eval. Let $N_k(j)$ be the $k$ instructions most similar to response $j$'s own instruction, excluding it, and $N_k(i)$ the $k$ responses whose instructions are most similar to $i$. The corrected score is
$$\tilde S_{ij} = 2S_{ij} - \frac{1}{k}\sum_{i' \in N_k(j)} S_{i'j} - \frac{1}{k}\sum_{j' \in N_k(i)} S_{ij'} .$$
The first average measures how much the judge likes response $j$ regardless of fit, which we call its hub score. The second measures how lenient the judge is on instruction $i$. We claim three things. First, judge score matrices show hubness: a heavy-tailed distribution of how often a response lands in the top-$k$ for instructions it was not written for, and this is concentrated on long, generic, or cue-bearing responses. Second, the correction improves agreement with humans on dimensions that are defined relative to the input (relevance, consistency, instruction-following), and it hurts on dimensions that are intrinsic to the response (fluency). That split is a falsifiable signature of the mechanism, not a nuisance. Third, the correction reduces verdict shifts from Silent-Judge-style cues without being told the cue exists.
The choice of near-miss neighbors matters. Randomly mismatched instructions give almost every response a near-zero score, so they carry little information. The hub signal lives in topically adjacent instructions, where a generic answer is almost plausible. Finding those neighbors is an embedding retrieval problem, and it is itself exposed to hubness: raw cosine search in sentence-embedding spaces returns the same generic items for many queries. So the neighbor sets are retrieved with the ratio-margin criterion. That is the second, supporting use of local scaling in this proposal.
Relation to Prior Work
The method comes from two sources. CSLS itself, from Unsupervised Multilingual Word Embeddings, penalizes a translation candidate by its average similarity to its neighbors. Margin scoring from Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings applies the same idea to sentence mining. Both operate on cosine similarities between vectors. We keep their arithmetic but replace cosine similarity with a judge's verdict, which is not symmetric and not a metric. Whether the correction still helps under those conditions is part of the question.
Three papers motivate the proposal by showing that density effects matter in evaluation. GICDM shows that hubs inflate distance-based image-generation metrics, and that correcting them raises Pearson correlation with human judgment on CIFAR-10 from 0.82 to 0.95. It anchors its corrections on real data and computes them out-of-sample. We adopt that discipline: neighbor sets come only from a fixed reference pool, so a response's corrected score does not depend on which other responses happen to be in the batch. GICDM works on image and audio embeddings, not on judge outputs. Hubness Reduction Improves Sentence-BERT Semantic Spaces documents text-embedding hubs, including whitespace strings acting as universal neighbors. It also warns that correction can hurt spaces with little hubness. We therefore measure hubness before correcting and report where it is absent. Measuring Data Diversity for Instruction Tuning shows that density-aware distances predict judge-measured quality far better than raw distances, but it treats the judge as a fixed black box.
On the judge side, the competing debiasing approaches name their confounders. Length-Controlled AlpacaEval fits a regression on length and instruction difficulty. The Silent Judge measures cue-induced verdict shifts but offers no fix. Calibrating LLM-Based Evaluator and G-Eval improve the prompt. Our correction is orthogonal to prompt design and can stack on either. It differs from the length regression in that it removes an estimated instruction-independent component rather than a pre-specified feature.
The obvious combination of these two fields would be margin-based retrieval of few-shot demonstrations for a judge, extending Towards Robust In-Context Learning for Machine Translation to evaluation. We treat that as a competing arm rather than the contribution. It changes what the judge sees; it does not measure or remove the judge's instruction-independent preference.
Research Program
The first stage is diagnostic. For each benchmark we build a partial score matrix: every response is scored against its own instruction and against $k$ margin-retrieved near-miss instructions, with $k$ swept over 5, 10, and 20. From this we compute the skewness of how often each response appears in the top-$k$ of instructions it was not written for, the same k-occurrence statistic used for embedding hubness. We then regress each response's hub score on length, formatting, and injected cues to see what the hubs are made of.
The second stage applies the correction and a family of variants, described under Baselines. The third stage runs the out-of-sample version: the neighbor pools and leniency terms are fixed from a reference split, and held-out responses are corrected independently. This makes the method deployable for a single new response.
Judges are open-weight models with token log-probabilities (Qwen2.5 and LLaMA-3 family, at 7–8B and roughly 70B scale). This follows from G-Eval's dependence on log-probabilities, which closed APIs increasingly restrict. It also lets us report cost honestly. The correction adds $k$ judge calls per response, so every comparison is run at matched extra calls.
Baselines
The do-nothing reference is the raw probability-weighted G-Eval-style score from the same judge and prompt. On SummEval, G-Eval reports a Spearman correlation of 0.514 for GPT-4. We replicate this protocol with our open judges and compare against our own replication, citing 0.514 only as context.
The strongest existing debiasing method is the length-controlled regression from Length-Controlled AlpacaEval. That paper reports that the correction raises Spearman correlation with Chatbot Arena from 0.94 to 0.98. We refit its model on our judge's AlpacaEval verdicts, using the same 805 instructions and the same model outputs. On AlpacaEval, beating it means matching 0.98 without a length feature, or exceeding the length-controlled result when our correction is stacked with it. The second outcome is the more realistic success criterion. On Silent-Judge cue tests, beating it means a lower verdict shift rate, since length control has no mechanism to address provenance cues.
The strongest prompt-side competitor is an AutoCalibrate-style induced rubric (Calibrating LLM-Based Evaluator), built from the same small golden set our reference pool draws from. We test it both as an alternative and stacked with our correction. The obvious-combination arm retrieves few-shot judge demonstrations by ratio margin, following Towards Robust In-Context Learning for Machine Translation with $k=10$, and is given the same extra-call budget.
The compute-matched baseline spends the same $k$ extra calls per response on ensembling instead: position swaps, several sampled chain-of-thought rationales, and several prompt paraphrases, then averages. If that ensemble matches our correction, the gain is just variance reduction.
Four ablations isolate the key ingredient. The first uses random rather than near-miss background instructions, testing whether locality matters. The second uses a global mean over all background scores rather than the top-$k$ average. The third drops the instruction-side leniency term. The fourth retrieves neighbors by raw cosine rather than by margin, testing whether embedding hubness contaminates the background.
Why not just…?
Why not just regress out length and known cues, as length-controlled AlpacaEval does? Regression works only for confounders someone thought to measure, and the Silent Judge results show judges respond to cues they never mention (0% acknowledgment). Our diagnostic regression of hub scores on length tests this directly. If length explains nearly all hub variance and the corrected score adds nothing beyond the length-controlled baseline, the regression is sufficient and we will say so. The cue-injection experiments are where the two approaches should separate.
Why not just z-score each response against random other instructions? This is the most serious threat, because it is simpler and cheaper to implement. Our argument is that random mismatches floor out for every response and so carry no hub signal, while near-miss instructions reveal it. The random-background and global-mean ablations settle this. If they match the local version, the contribution shrinks to "instruction-independent residualization," and locality is dropped from the claim.
Why not just ask for chain-of-thought or a better rubric? The Silent Judge found that judges asked to justify their verdicts still shifted by up to 30% on cues while never citing them. Calibrating LLM-Based Evaluator found chain-of-thought adds little once a good rubric exists. Rubrics remain a real competitor on correlation, so we stack rather than replace them and report both.
Why not just spend the extra calls on an ensemble? Averaging reduces noise but keeps any bias shared across samples. A preference for long or expert-labeled answers is shared. The compute-matched ensemble baseline measures exactly this difference.
Proposed Experiments
On SummEval, a summary of one article is scored against near-miss articles, which gives a clean mismatch test. We report per-dimension Spearman correlation with expert labels. The decisive check is the predicted split: gains on relevance and consistency, no gain or a loss on fluency. We test this on SummEval and Topical-Chat, and on QAGS for hallucination detection. A uniform gain across all dimensions, or a uniform loss, would count against the mechanism.
On AlpacaEval, we correct pairwise judge preferences across the 805 instructions and compute system-level Spearman correlation with Chatbot Arena against the 0.94 and 0.98 reference points.
On ELI5 and LitBench, we reproduce the Silent Judge's provenance and recency injections. We measure verdict shift rate before and after correction, and check whether cue-bearing responses rise in hub score. We expect a large drop from the +30% recency shift reported for GPT-4o on ELI5 with open judges under the same protocol. We report whatever shift our open judges show.
Across all settings we report hubness skewness alongside the gains. Following the Sentence-BERT warning, we expect benefits only where judge hubness is present, and we treat low-hubness datasets as a control. The correction should leave them roughly unchanged. If those datasets degrade noticeably, we will add a skewness gate before applying the correction.
References
- Unsupervised Multilingual Word Embeddings
- Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
- GICDM: Mitigating Hubness for Reliable Distance-Based Generative Model Evaluation
- Hubness Reduction Improves Sentence-BERT Semantic Spaces
- Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
- Length-Controlled AlpacaEval: A Simple Debiasing of Automatic Evaluators
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Calibrating LLM-Based Evaluator
- G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
- Towards Robust In-Context Learning for Machine Translation with Large Language Models