Teams that run retrieval-augmented generation (RAG) systems, where a language model answers questions from documents fetched out of a search index, increasingly use another language model as the grader. Human review cannot keep up with every model or index update. The grading model, often called an LLM judge, reads the retrieved documents and a candidate answer, then decides whether the answer is supported. We hypothesize a specific failure in this setup. When the supporting passage sits in the middle of a long stack of documents, the judge does not simply get noisier. It switches to a different decision rule: it rewards answers that sound familiar, meaning common words and frequent entities, over answers that the documents actually support.
Two known findings make this plausible. Language models often do worse than having no documents at all when the key fact is buried mid-context (Lost in the Middle). Separately, reference-free evaluators largely score text by how well it matches their own internal language-model distribution (On the Limitations of Reference-Free Evaluations of Generated Text). If both hold for judges, a judge facing middle-buried evidence behaves like a closed-book frequency scorer. That matters most when an index is updated and the new retriever surfaces long-tail documents or reorders the evidence. In that case the automated verdict on "did the upgrade help?" may be measuring document order and vocabulary frequency rather than answer quality. We propose to measure this interaction directly and to correct it with a cheap contrastive scoring rule. The rule subtracts the judge's no-context opinion from its with-context opinion.
Core Claim
The claim has two parts. First, frequency reliance in LLM judges depends on position. If we regress a judge's verdict on two variables, whether the answer is supported and how much more frequent its phrasing is than the alternative's, the frequency coefficient should be small when the gold evidence is at the start or end of the context. It should grow sharply when the evidence is in the middle. In the middle, the judge's verdicts should agree more with the same judge run closed-book than with ground truth.
Second, removing the closed-book component fixes most of this at roughly twice the cost of one judge call. The fix is to score the judge's log-odds of "supported" with documents, minus a weighted copy of the same log-odds without documents. Running the judge over many document orderings would cost many times more.
The existing work leaves this open in three places. Judging the Judges showed that judges fall back on presentation order when the quality gap between candidates is small, but it used short pairwise prompts and never varied evidence depth. LLMs instead of Human Judges? documented that judges rely on surface fluency and do worse on machine-generated text, but only on short inputs. Judge as A Judge (ConsJudge) trains RAG judges to agree with themselves across rubric prompts, yet never asks where the evidence sat or whether the preferred reasoning favors frequent phrasing. A judge that consistently relies on a frequency prior would score as perfectly consistent under that training signal.
Research Program
Diagnostic benchmark. We build the benchmark on the multi-document question-answering setup of Lost in the Middle: one gold Wikipedia passage among 10, 20, or 30 distractors, with gold position varied across all slots. We use Natural Questions, TriviaQA, and HotpotQA as in the ConsJudge suite.
For each question we build two candidate answers in a two-by-two design. One axis is support: the answer is supported by the gold passage or not supported by any passage. The other axis is frequency: the answer is the more frequent or less frequent of the two. Frequency is measured two ways, by corpus frequency of the answer entity and by the judge's own closed-book log-probability of the answer. Unsupported answers are same-type entities that are plausible but absent from every document.
The design is necessary because frequency and correctness are usually correlated. Only the cells where the supported answer is the rare one can separate "the judge checks evidence" from "the judge prefers familiar answers." Short-answer QA has gold labels, so correctness is known without human annotators. This is what makes a dense sweep over positions affordable.
Measuring frequency reliance. We fit a logistic model of judge verdicts on support, frequency gap, gold position, and their interactions. The frequency-reliance curve is the frequency coefficient as a function of position. We report it alongside two other quantities: the judge's accuracy curve, and its agreement with the closed-book judge at each position.
We also borrow the acknowledgment measurement from The Silent Judge. For each rationale we check whether it quotes the gold passage. That work found judges never admit the shortcut that drove their verdict, so we expect rationales for middle-position errors to cite fluency or plausibility instead of evidence. If so, reading the rationale cannot detect the failure.
Contrastive judging. We read the judge's verdict from token log-probabilities on fixed answer tokens, following the final step of EvalAssist. Let $\ell(c, a)$ be the log-odds of "supported" for answer $a$ given documents $c$, and let $\ell(\varnothing, a)$ be the same quantity with no documents. The corrected score is
$$ s(a) = (1+\beta)\,\ell(c, a) - \beta\,\ell(\varnothing, a), $$
where $\beta \ge 0$ is tuned on a held-out split. This mirrors the decoding-time correction in Positional Contrastive Decoding, which subtracts a "near-sighted" baseline from the model's predictions. Our baseline is instead the judge's parametric prior. That choice follows from the result in On the Limitations of Reference-Free Evaluations that such scores mostly track an internal pseudo-reference.
Application regime: retriever upgrades. We compare three retrieval setups over one fixed text corpus:
- an old dense encoder;
- a new encoder with the whole index re-encoded (full backfill);
- a new query encoder trained to search the old index directly, using the influence loss from Towards Backward-Compatible Representation Learning, with the orthonormal-rotation variant of BT² as a second compatible option.
Each setup feeds the same generator. We then ask whether a judge-based decision about which retriever to ship matches the ranking given by gold answers. We add NFCorpus, the medical retrieval set used by AURORA, where new rare terminology is the point of the update. On NFCorpus we collect expert annotations on a stratified subset of a few hundred long-form answers.
The specific hypothesis is that compatible retrievers change both the order of evidence and how often rare-term documents are retrieved. The backward-compatibility papers report a 1–3% accuracy cost relative to a fully re-indexed model on face search in Shen et al., but no comparable figure exists for text RAG. A frequency-reliant judge could therefore misrank the systems. It might credit or blame the compatibility constraint for effects that come purely from evidence position.
Baselines
Naive judge. The reference point is a single pointwise call with documents in retriever order, and with chain-of-thought reasoning before the verdict. This is how most RAG pipelines are graded today. Without a correction, contrastive judging should show a flat frequency-reliance curve.
Order randomization. The strongest direct competitor shuffles the documents $k$ times, judges each ordering, and averages the log-odds. It extends the position-swap check of Judging the Judges from two candidates to document order. Because it averages over positions, the middle penalty is spread out rather than removed, which is why it is the threat to take seriously. We run it at $k = 2$, which matches our cost, and at $k = 8$. All methods use the same judge models (an open-weight 8B–14B judge and one frontier API judge), the same prompts, and the same dev split for tuning $k$ or $\beta$. Beating it means two things. First, at matched cost, contrastive judging should have higher accuracy in the rare-supported cells. Second, at $k = 8$, contrastive judging should come within the noise band of the averaging baseline while costing a quarter as much. If $k = 2$ averaging matches us, our method has no advantage, and we will report that.
Positional Contrastive Decoding. The strongest published method is PCD applied to the judge's verdict tokens (Xiao et al.). PCD reports variable-tracking F1 on RULER 16k rising from 64.40 to 69.04, and InfiniteBench 8k key-value retrieval rising from 72.0% to 79.0% on Llama-3-8B-262k. We found no result on Lacuna for PCD or order averaging applied to LLM judges. The nearest judge numbers are a position consistency of about 0.82 for GPT-4 and Claude-3.5-Sonnet in Judging the Judges, and judge consistency above 0.90 for ConsJudge. We also train a ConsJudge judge on our data and use it as a trained-judge baseline.
Ablations. Three ablations isolate the key ingredient. The first sets $\beta = 0$. The second replaces the closed-book term with a "shuffled-irrelevant-documents" term, which tests whether what matters is the parametric prior or any change in context. The third applies the correction only to edge-position items, where it should do nothing.
Why not just…?
Why not just shuffle the documents and average? This is the real threat, and the matched-cost comparison above will settle it. Our hypothesis is that averaging dilutes frequency reliance but does not remove it. If the middle positions push every ordering toward the same familiar answer, averaging keeps that bias. Contrastive judging targets the shared cause. If $k$-shuffle averaging closes the gap at $k = 2$, we will treat the diagnostic curve as the main contribution and recommend averaging.
Why not just rerank so the best evidence sits at the edges? The judge does not know which passage is the evidence, and reordering by retriever score depends on the very retriever we are trying to evaluate. On multi-hop HotpotQA, two required passages cannot both occupy the first and last slots. RULER and Lost in the Middle both note this forced use of middle positions. We include retriever-order-to-edges as a baseline and report it separately for single-hop and multi-hop questions.
Why not just use a stronger long-context judge with chain-of-thought? Lost in the Middle found that long-context-tuned variants show the same U-shaped accuracy curve. LLMs instead of Human Judges? found chain-of-thought does not consistently improve agreement with expert humans. The Silent Judge shows rationales hide the shortcut that drove the verdict. We test the frontier judge anyway. A flat frequency-reliance curve for that judge would mean the problem is confined to small judges.
Why not just split the documents into chunks and check support chunk by chunk? That design avoids long contexts entirely, but it costs one call per chunk and cannot verify multi-hop answers whose support spans chunks. We include it as a cost-accuracy reference point on single-hop data.
Decisive Experiments
The first experiment tests the hypothesis itself. In the rare-supported cells, does judge accuracy at middle positions fall to or below closed-book judge accuracy? Does the frequency coefficient rise in the middle, while it stays near zero in the frequency-balanced cells? If the frequency coefficient is flat across positions and only accuracy drops, the frequency-reliance hypothesis is wrong and the middle curse is plain noise. We would report that negative result, and the contrastive method would lose its rationale.
The second experiment tests the fix. Contrastive judging should cut the middle-position accuracy deficit in rare-supported cells by at least half relative to the naive judge. It should also beat two-shuffle averaging at equal cost on Natural Questions, TriviaQA, and HotpotQA, and leave edge-position accuracy unchanged.
The third experiment tests the application. On the three retriever setups, we compare system rankings from the naive judge, the contrastive judge, and gold answers or expert annotations on NFCorpus. A misranking by the naive judge that disappears under contrastive judging would show that automated upgrade decisions for re-indexing are currently confounded by evidence position and vocabulary frequency. If no misranking appears, the practical stakes are smaller than we claim, and we will say so.
References
- Lost in the Middle: How Language Models Use Long Contexts
- On the Limitations of Reference-Free Evaluations of Generated Text
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
- LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks
- Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- EvalAssist: LLM-as-a-Judge Simplified
- Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
- Towards Backward-Compatible Representation Learning
- BT: Backward-compatible Training with Basis Transformation
- AURORA: Neuro-Symbolic Continual Indexing for Evolving RAG Systems
- RULER: What’s the Real Context Size of Your Long-Context Language Models?