When a retrieval-augmented system answers a question, an LLM judge (a large language model prompted to grade another model's output) is often asked a narrow question: is this answer supported by the passages the system was given? That question has a right answer that does not depend on whether the claim is true in the world. This proposal tests a simple hypothesis. A judge that has memorized the source document during pre-training will mark claims as "supported" when the retrieved passages do not actually support them, because the judge fills the gap from memory. The failure should be worst exactly where retrieval pipelines are weakest. Documents are cut into chunks, and the chunk carrying the key fact, its section header, or its cross-reference is often left out. The judge sees an incomplete record, remembers the rest, and grades as if the record were complete.
The contribution is a measurement method and a causal test, not another benchmark of judge bias. We use prefix probing to estimate, document by document, how much of each source text the judge itself has memorized. Prefix probing means feeding a model the opening of a text and checking whether it reproduces the continuation verbatim. We then relate that per-document memorization dose to the judge's rate of wrongly certifying unsupported claims. We confirm the relationship causally by fine-tuning memorization into an open judge on half of a set of fresh documents. If the hypothesis holds, faithfulness scores on public long-document benchmarks are partly a measure of what the judge read during training. Groundedness numbers reported on those benchmarks would be inflated in a way that no current audit detects.
Core Claim
We claim that for retrieval-grounded faithfulness judging, a judge's verbatim memorization of the evidence document predicts, and causally increases, its false-support rate. The false-support rate is the fraction of claims that are true in the full document but absent from the supplied chunks, yet are judged supported. We further claim that this rate rises when the supplied chunks are fragmented, meaning headers and neighboring context are stripped, and when distracting passages are added. Both conditions push the judge to rely on what it already knows rather than on what it was shown.
This sits in a gap that existing judge-bias work leaves open. The Silent Judge shows that GPT-4o and Gemini-2.5-Flash shift verdicts on surface cues such as recency labels (a verdict shift rate of up to +30% on ELI5). It also reports a cue acknowledgment rate of 0%: the judges never admit the cue drove them. That paper motivates us, because a judge grading from memory would likewise never say so. Its cues, however, are injected metadata tags, not the judge's own training data. Preference Leakage finds that judges inflate win rates for models in their own family (Preference Leakage Score up to 37.1% on Arena-Hard). It attributes this to stylistic mimicry rather than to memorized content, and it studies short conversational turns rather than grounded evaluation. RADAR detects whether a candidate model is recalling rather than reasoning, reaching 93.0% accuracy from internal features. It audits the model being evaluated, never the evaluator, and uses only single clean prompts.
One paper appears to cut against us. On the Limits of LLM Adaptability finds that text-memorization measures correlate negatively with annotation accuracy (r = -0.19), while conceptual familiarity with the task definition predicts it (partial r = +0.41). We take this seriously but do not see it as a contradiction. That study measured string similarity between model outputs and rubrics, not extraction of the items' source documents. It also studied toxicity labeling, where no context-grounding constraint exists. Our hypothesis concerns a different kind of memory, memory of the evidence, and a different error, leaking it into a judgment that should be context-only. The same paper's "decision stickiness" finding constrains our design. Few-shot corrections rescued only 34.8% of zero-shot errors, and only 20.8% when the judge was highly confident. We therefore expect instruction-level fixes to be weak, and we test them directly instead of assuming it.
Research Program
The first component is a memorization meter for the judge, built to avoid known probe failures. Probe Choice Changes Canary-Memorization Verdicts shows that averaging likelihood over a fixed 20-token window can flip verdicts. In one case, 99% of the metric's movement came from non-secret preamble text, and in another the probe missed a drop in hit@1 from 100% to 88%. We take that paper as a hard constraint on method. For each source document we score only the content span that follows each prefix, at several prefix offsets. We report exact-continuation hit@1 on held-out spans rather than a windowed likelihood average. Following Schwarzschild, we add a compression-style score for open models: how short a prompt reconstructs a long span. For closed API judges, whose log-probabilities are hidden, we use the multiple-choice discrimination format of DE-COP (Duarte et al.). The judge chooses the verbatim passage among paraphrases. NumLeak warns that greedy decoding under-reports memorization visible in log-probability rankings, so on open models we report both. Each document ends up with a memorization dose for each judge.
The second component is a grounding test set built by controlled fragmentation. We start from long-document QA items with annotated supporting evidence, drawn from UDA (UDA) and Loong (EnSI-RAG). We turn each gold answer into a declarative claim. We chunk the document with fixed-length splitting and retrieve the top passages with a standard retriever, so the context set looks realistic. We then produce three versions of each item. The supported version keeps the gold evidence chunk. The removed version drops the gold chunk and keeps the near-miss chunks. The detached version keeps the evidence sentence but strips the section header or the cross-referenced section that qualifies it. This last version follows the structural-disconnection failure in Does Structure Matter? and the dependency severing described for technical standards in Erak et al.. In the removed and detached versions the correct verdict is "not supported by the given context," and human annotators verify a sample of every version. A noise arm adds distractor passages: random off-topic chunks, plus single adversarial sentences found by the error-guided tree search of Adaptive Distraction. That method lowered frontier-model accuracy by over 45%, against under 5% for static insertions.
The third component is the causal test, which carries the main weight of the claim. Natural memorization is confounded with document popularity and difficulty: a heavily memorized Wikipedia page is also an easy, well-known page. We therefore collect documents published after the open judges' training cutoffs, such as recent filings, papers, and reports. We split them at random, and LoRA-fine-tune an open judge on the full text of one half, using the repetition levels needed to reach high hit@1. We use a matched-norm random-adapter control (the LORA-NOISE control from the canary testbed), and we check that general judging accuracy on unrelated grounding items is unchanged. Because the split is random, any difference in false-support rate between the memorized and unmemorized halves is caused by memorization.
The fourth component is a diagnostic that isolates memory without needing training-data access. We re-run each item after consistently renaming the document's distinctive entities and numbers, in both the context and the claim, to fictional values. This borrows the fictional-name control NumLeak uses to separate recall from fabrication. Renaming leaves the grounding question unchanged but disconnects parametric memory. A verdict that flips from "supported" to "not supported" after renaming is direct evidence of memory leakage. The disagreement rate also gives a practical audit signal for closed judges.
Baselines
The naive reference point is the standard faithfulness prompt run on each judge as is: Llama-3.1-8B and 70B, Qwen-2.5-7B and 14B, GPT-4o, and Gemini-2.5-Flash. These are graded on the post-cutoff, unmemorized documents, where no memory can leak. This sets the floor false-support rate attributable to fragmentation alone. Any excess on memorized documents is the effect we claim.
The strongest prompt-level competitor is a strict grounding instruction: "judge only from the passages given; ignore anything you know." This is paired with a requirement that the judge quote the supporting span before giving a verdict. It is strongest because NumLeak reports that a one-line system prompt blocked 99.8% of direct recall attempts, so an instruction may well suppress memory leakage too. We run it with the same items, the same decoding, and an equal budget of prompt variants tuned on a held-out development split. The quote requirement is checked mechanically: a quoted span that does not appear in the context counts as a failure.
The strongest calibration competitor adapts the contextual calibration of Preference Leakage. It reports reducing that paper's error bias from 17.8 to 7.3. Here we estimate the judge's prior toward "supported" using content-free context and subtract it. Its weakness is that our bias depends on the document, while calibration removes only a global offset. The experiment will show whether that matters.
A memory-free reference is a natural-language-inference checker: a smaller model that decides whether the passages entail the claim, in the spirit of the entailment approach of Dhruva et al.. It has far less capacity to have memorized the sources, but it is a weaker reasoner over long, fragmented evidence.
The ablation isolating our key ingredient compares the entity-renamed re-judging check against the same pipeline with the renaming replaced by a meaning-preserving paraphrase that keeps the real entities. If paraphrase works as well, then the effect comes from surface re-wording rather than from cutting the link to memory.
No reported result was found on Lacuna for false-support rates of LLM faithfulness judges under chunk removal, so there is no published figure to beat. The closest reported numbers concern different failures: a verdict shift of +30% from recency cues, a 0% cue acknowledgment rate, a leakage score of 37.1%, and calibration reducing error bias from 17.8 to 7.3. We count our diagnostic as beating a baseline only under two conditions. First, on the causal LoRA split it must reduce the memorized-minus-unmemorized false-support gap more than the baseline does. Second, it must not cut recall on supported items by more than two points, since a judge that marks everything unsupported trivially wins.
Why not just…?
Why not just tell the judge to use only the context? This is the most serious threat, because NumLeak's one-line instruction blocked 99.8% of direct recall. That result, however, concerns stopping a model from emitting memorized values on request. Our failure is different: memory silently filling a gap during a judgment. The Silent Judge's 0% acknowledgment rate and the decision stickiness in the annotation study suggest the judge may not know it is relying on memory. Adaptive Distraction also found that "ignore irrelevant information" instructions gave almost no protection against distractor passages. The strict-instruction baseline on the LoRA split settles the question. If it closes the memorized-unmemorized gap, our main recommendation becomes "use this instruction," and the measurement method remains valuable as the audit that showed it was needed.
Why not just compare pre-cutoff and post-cutoff documents without any probing? This comparison is cheap, but it confounds memorization with popularity, writing style, and difficulty, all of which differ systematically between old and new documents. The randomized LoRA split removes that confound. The per-document probe dose, used in natural models, gives a dose-response curve within a single time period that a binary date split cannot provide.
Why not just give the judge the full document instead of the chunks? That changes the question. The judge would assess whether a claim is true in the document, not whether the generator had evidence for it, and the second question is what faithfulness evaluation exists to answer. Full-document judging also brings its own collapse: UDA reports long-context GPT-4 scoring 0.4 on FinHybrid against 45.9 with retrieval, and HELMET finds steep drops for open models toward 128k tokens. We still report full-document judging as a side condition, for completeness.
Why not just use a small entailment model that never memorized anything? This substitutes one error for another, and the NLI baseline measures the cost directly. If its recall on supported fragmented items stays within a few points of the LLM judges while it avoids memory leakage, that is a useful practical result, and we will report it as such.
Proposed Experiments and Decisive Checks
The primary experiment is the randomized LoRA split on post-cutoff documents, run with two open judge sizes. It succeeds if the false-support rate on removed and detached items rises on the memorized half relative to both the unmemorized half and the matched-norm noise adapter. The rise should grow with repetition count and leave supported-item recall unchanged. A null result here, with probe hit@1 confirmed high, falsifies the core claim.
The second experiment estimates the natural-model dose-response across UDA's FinHybrid, PaperText, and NqText subsets and Loong, for all six judges. False-support rate is regressed on probe dose, controlling for question difficulty as measured by the unmemorized-judge error rate and for document length. NqText draws on world knowledge, so we expect high memorization doses there, while recent papers should give low doses.
The third experiment crosses dose with the noise arm. We predict that adaptive distractors increase false support more on high-dose documents. A noisy context would then push judges further onto memory, which is the interaction that gap analyses of this literature leave unmeasured.
The fourth experiment validates the renamed re-judging diagnostic as an audit. On the LoRA split, where ground truth about memorization is known, we ask whether disagreement between original and renamed verdicts identifies memorized documents. We then apply the diagnostic to GPT-4o and Gemini-2.5-Flash, where memorization can only be inferred. For this step we report agreement between the diagnostic and the DE-COP-style probe dose.
We also measure acknowledgment: we ask, in a fraction of cases, how often a judge's written justification cites content that is absent from the context. This tests whether the silent rationalization The Silent Judge found for injected cues also appears for memory.
References
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
- Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbed
- Rethinking LLM Memorization through the Lens of Adversarial Compression
- DE-COP: Detecting Copyrighted Content in Language Models Training Data
- NumLeak: Public Numeric Benchmarks as Latent Label in Foundation Models
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- EnSI-RAG: Entity-Structure-Indexed Retrieval-Augmented Generation for Long-Document Question Answering
- Does Structure Matter? Encoding Documents for Machine Reading Comprehension
- Leveraging Fine-Tuned Retrieval-Augmented Generation with Long-Context Support: For 3GPP Standards
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Solving Open-Domain Multiple Choice Questions with Textual Entailment and Text Similarity Measures
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly