Modern retrieval benchmarks can no longer afford human assessors for every document a new search system might return. Evaluators therefore ask a large language model (LLM) to fill in the missing relevance labels. The recent consensus is that a small model fine-tuned on human judgments does this job well. This note argues that such a judge has a failure mode nobody has measured. It grades a passage partly by whether the passage agrees with facts it memorized during pre-training, rather than only by whether the passage addresses the query. If that is true, training-data contamination does not just inflate model scores. It gets written into the benchmark's answer key. Retrievers that surface current or unfamiliar evidence are then penalized, and so are retrievers whose content diverges from what the judge "knows."
The opportunity sits where three known problems meet. Binary relevant/irrelevant labels hide gradations of usefulness. Pooling bias means only documents found by the original participating systems get judged, so everything else is silently scored as irrelevant. Benchmark contamination means models have seen test material before. The proposal treats pooling as the regime (LLM judges exist to fill pooling holes), contamination as the mechanism (the judge's own memory), and graded relevance as the lens. The central prediction is that memory-driven bias hides in intermediate grades, where binary agreement statistics cannot see it.
Core Claim
An LLM relevance judge should give the same graded relevance to two passages that answer a query equally directly but state different values, for example the pre-edit and post-edit version of a Wikipedia fact. Correctness and currency are a separate axis from relevance. We hypothesize that fine-tuned and prompted judges violate this invariance. The size of the violation should grow with how confidently the judge can answer the query without any document (closed-book), and it should be larger on 0–5 graded outputs than on binary outputs. We further claim three things. The violation is enough to reorder system leaderboards when the judge re-labels an entire pool. Instructing the judge to "use only the document" does not remove it. A cheap test-time correction removes most of it: score the passage together with entity-swapped counterfactual copies and average.
The closest prior work motivates and constrains this claim. Can We Use Large Language Models to Fill Relevance Judgment Holes? fine-tunes Llama-3-8B to fill unjudged documents on TREC iKAT 2023. It shows that the entire pool must be re-judged by the LLM for consistency. That recommendation makes any systematic judge bias pervasive rather than local, and the paper never tests memory-dependence. DIRAS distills GPT-4 relevance definitions into calibrated small judges that catch human false negatives more than 90% of the time. Its calibration is measured on static corpora where judge memory and document content rarely conflict. Augmenting Researchy Questions with Sub-question Judgments produces 24 million graded LLM labels without human verification of systematic judge error. It is exactly the kind of resource our audit is meant for. On the contamination side, Search-Time Data Contamination shows that models override their own reasoning when an answer key is visible. HoH shows that generators are misled by outdated passages. INVITE shows that open models accept distorted premises. All three study the model being evaluated. None studies the model doing the evaluating, which is our focus.
Research Program
The first step builds matched passage pairs that differ only in the stated fact. Three sources supply them. HoH already pairs current and outdated evidence extracted from Wikipedia token-level diffs. LiveSearchBench supplies post-cutoff Wikidata changes whose new values the judge cannot have memorized. D-MERIT provides exhaustively annotated Wikipedia evidence, which we perturb with INVITE-style entity swaps. For each pair we first probe the judge closed-book, recording whether it answers the query and how much probability it gives the answer. This lets us sort pairs into "judge knows value A," "judge knows neither," and "judge knows the superseded value."
The second step measures the memory-agreement gap. This is the mean graded score a judge gives the passage matching its closed-book answer, minus the score it gives the counterfactual twin. We measure it for binary and 0–5 outputs, and for direct scoring versus rubric-with-reasoning scoring in the style of Retro*. A small human study of about 1,500 pairs, graded on the same rubric, establishes that humans are close to invariant. It also separates legitimate cases where a swapped value makes a passage incoherent.
The reasoning-versus-direct comparison matters for a specific reason. Beyond Polarization shows that reasoning chains compress continuous relevance into coarse modes. We hypothesize an additional channel: a reasoning chain gives the judge room to recall the answer and check the passage against it.
The third step measures consequences. We simulate the whole-pool re-judging protocol. Pools are built from several retrievers (BM25, a dense retriever, SPLADE-style sparse). We then add controlled "fresh-preferring" and "stale-preferring" rankers, constructed by reordering candidates according to HoH's current/outdated labels. Each pool is re-judged by each LLM judge. We report Kendall's tau of the resulting system ranking against rankings from human or dataset-provided labels, along with the rate at which pairs of systems swap order.
The fourth step tests two corrections. The first is test-time swap-averaging: score the original passage and k entity-swapped copies with the query's answer slot varied, then average, so the judge's memory has no single value to latch onto. The second is fine-tuning the judge with a swap-consistency penalty on top of the recipe from the hole-filling paper.
Baselines
The naive reference point is the closed-world assumption with no hole filling: unjudged documents count as irrelevant. D-MERIT quantifies what this costs. Standard pooling captures about 47% of evidence, and system-biased partial annotation produces a 19.2% system-ranking error rate. Any judge must beat this ranking error to justify its use at all.
The strongest existing competitor is the fine-tuned Llama-3-8B-Instruct hole-filler from Can We Use Large Language Models to Fill Relevance Judgment Holes?. It reports binary Kappa of 0.729 against human labels on TREC iKAT 2023, versus 0.41–0.48 for zero-shot GPT-3.5, and system-ranking Kendall's tau often above 0.80. We will reproduce it with its QLoRA recipe on iKAT to confirm we match that Kappa before auditing it. We then use the identical checkpoint on our pairs. This competitor is strongest because it is the protocol the field currently recommends.
The second strong competitor is a DIRAS-style calibrated token-probability judge. DIRAS reports a calibration score of 91.35 for Llama3-Tok against 91.10 for GPT-4 on its climate retrieval data. It is the most calibrated option, so it is the fairest test of whether calibration alone protects against memory bias. The third is a Retro*-style multi-sample rubric reranker used as a judge, since Retro* reports 36.6 average nDCG@10 on BRIGHT at 7B.
All judges share the same base model family where possible, the same training labels, the same pair sets, and a matched inference budget. Swap-averaging with k copies is compared against Retro* with k sampled reasoning trajectories, so extra compute is not the explanation for any gain.
The key ablation isolates the ingredient. Swap-averaging with random, answer-irrelevant entity swaps (touching non-answer spans) controls for "averaging over perturbations." Swap-averaging on the answer slot tests "removing the memorized value." Beating the strongest baseline means two things together: reducing the memory-agreement gap by at least half, and raising leaderboard Kendall's tau against human labels on the fresh/stale pools, without dropping iKAT Kappa below the reproduced 0.729.
Why not just…?
Why not just instruct the judge to rely only on the document? This is the cheapest fix and a real threat. We run it as an explicit baseline, both as a prompt instruction and as part of the fine-tuning data. The evidence against it is indirect but pointed. Search-Time Data Contamination documents reasoning models explicitly overriding their own correct reasoning once an answer is available. HoH finds even GPT-4o unreliable at recognizing superseded evidence. If the instructed judge's gap is within noise of the human gap, the correction is unnecessary, and the proposal reduces to an audit showing that instructions suffice. We would report that plainly.
Why not just use a newer frontier judge with a later training cutoff? A later cutoff moves the boundary but does not remove it. Every fact the newer judge has memorized can still conflict with a retrieved passage, and HoH-style superseded facts are exactly where an old value was seen many times. We include a prompted frontier judge and stratify results by closed-book confidence. If the gap tracks confidence regardless of model size or cutoff, newer models are not a solution. The hole-filling paper's finding that prompted GPT-3.5 reaches only 0.41–0.48 Kappa also suggests prompted frontier judges carry their own leniency bias.
Why not just evaluate only on post-cutoff dynamic benchmarks such as LiveSearchBench? Dynamic benchmarks keep the evaluated model from memorizing answers. They do not help with the judge, and they are the setting where the judge's memory of the old value most directly conflicts with new evidence. They also cannot cover stable-knowledge collections like D-MERIT, Researchy Questions, or BRIGHT, which the community continues to use. Our LiveSearchBench arm tests this directly. If judges show no gap on post-cutoff pairs where they know the superseded value, this alternative wins for that regime.
Why not just have humans re-judge a sample? Human spot checks estimate average agreement. But the bias we target is conditional on judge memory and concentrated in intermediate grades, so a random sample with binary scoring can show high agreement while leaderboards still shift. Our human study is designed around exactly this conditional stratification.
Decisive Experiments
The first decisive check is whether the memory-agreement gap exists and is monotone in closed-book confidence for the reproduced iKAT judge on HoH and D-MERIT swap pairs, while the human gap on the same pairs stays near zero. A flat or absent gap falsifies the core claim.
The second is whether the gap is larger on 0–5 outputs than on binary outputs while binary Kappa stays high. That would show that current agreement reporting cannot detect the bias.
The third is the leaderboard test. After whole-pool LLM re-judging, does the stale-preferring ranker move above the fresh-preferring ranker relative to their human-label order, and does the Kendall's tau loss exceed the 3.2% random-annotation error baseline from D-MERIT?
The fourth is whether answer-slot swap-averaging closes the gap and restores ranking fidelity where both the random-slot control and the "use only the document" instruction fail. If only the instruction is needed, we report that as the finding.
The result is either a concrete warning that LLM-filled test collections inherit their judges' pre-training memory, together with a cheap fix, or a clean negative result that licenses the current practice.
References
- Can We Use Large Language Models to Fill Relevance Judgment Holes?
- DIRAS: Efficient LLM Annotation of Document Relevance for Retrieval Augmented Generation
- Augmenting Researchy Questions with Sub-question Judgments
- Search-Time Data Contamination
- HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
- Evaluating D-MERIT of Partial-annotation on Information Retrieval
- Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval
- Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking