Graded Search-Time Contamination: Measuring How Much of an Answer Key a Retrieved Page Leaks

When a search-augmented language model is evaluated on a public benchmark, it can find the answers on the web. Current audits treat this as yes-or-no: a retrieved page either is a known mirror of the benchmark's answer file or it is clean. Most real leakage is partial. A forum thread may give the key intermediate result of a hard problem, a blog post may paraphrase half of a worked solution, or a course page may state the lemma the question was built around. We propose to score contamination the way modern retrieval research scores relevance, as a matter of degree. Each benchmark item's reference solution is broken into small checkable claims, and each retrieved page is scored by the fraction of those claims it supports. We then measure how much benchmark accuracy rises as this "leakage dose" grows.

The question matters because yes-or-no auditing may undercount the problem. Search-Time Data Contamination found that only about 3% of Humanity's Last Exam (HLE) items were reachable through HuggingFace mirrors. On those items, Sonar Deep Research scored 37.5% against 17.4% on the rest. If partial leaks that no URL filter catches produce even a fraction of that gain, the published "clean" numbers for search agents are also inflated. A retrieval-side audit also has a practical advantage over model-side detectors, which recent work shows can be erased by post-training: it only needs the pages the agent read, so it works on closed APIs.

Core Claim

The binary-relevance critique from information retrieval applies directly to contamination labels. A yes-or-no label throws away the gradient between "irrelevant" and "the full answer key," and it lets paraphrased or fragmentary leaks pass as clean. We claim three things. First, a graded leakage score derived from claim decomposition predicts per-item inflation in search-augmented agents. Second, on multi-step benchmarks such as HLE, a substantial share of total inflation comes from pages that URL matching and exact answer-string matching both miss. Third, the dose-response relationship has a measurable shape, either roughly proportional to coverage or threshold-like around a single "crux" step. That shape tells benchmark builders which leaks matter.

We define the leakage score of retrieved document $d$ for benchmark item $q$ as

$$\ell(d,q) = \frac{1}{|C_q|}\sum_{c \in C_q} p(d \models c),$$

where $C_q$ is the set of atomic claims in $q$'s reference solution (final answer plus intermediate steps), and $p(d \models c)$ is a judge model's probability that $d$ states or directly entails claim $c$. The item-level dose is the maximum, or the union coverage, of $\ell$ over everything the agent read before answering.

Relation to Prior Work

The measurement tool comes from the graded-relevance literature. Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability (Q-CARE) scores a retrieved chunk by the fraction of decomposed sub-queries it satisfies. On open-ended queries, its retrieval-precision scores correlate with human judgments at 0.55 Pearson, against 0.22 for the reference-based RAGChecker. Augmenting Researchy Questions with Sub-question Judgments similarly shows that multi-aspect information needs cannot be captured by one binary label. We reuse this decomposition idea but point it at a different target. Q-CARE measures whether a chunk helps answer a user's question. We measure whether a chunk gives away a benchmark's answer key, so the decomposition runs over the reference solution rather than the query.

The way we read judge probabilities follows Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking. That paper shows that making a model reason in text before emitting a relevance label degrades fine-grained ranking: Qwen3-32B scores 73.7 with direct scoring versus 70.1 with reasoning first on TREC DL19. We therefore read entailment probabilities directly from the judge's output distribution rather than from a reasoned verdict, and we test this choice as an ablation.

Prior contamination work both motivates and constrains the design. Search-Time Data Contamination established that search agents copy answer keys, but it defines contamination as retrieving a URL on a known benchmark mirror. Our work is the graded extension of that audit. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that membership-inference detectors fall to near-chance AUROC after brief reinforcement-learning post-training. RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation needs white-box activations. Both results argue for auditing what the agent read rather than what the model weights remember, and that is the only kind of contamination we address. Test of Time: Rethinking Temporal Signal of Benchmark Contamination shows that paraphrasing source material into reasoning problems erases surface signals: influence-function attribution falls from 77.5% to 17.5%. This is why our score must be robust to paraphrase, and why we test it on paraphrased leaks specifically.

The broader taxonomy in Benchmark Data Contamination of Large Language Models: A Survey already distinguishes levels from semantic to label exposure, but only for training data and only as categories. We provide a continuous measure for the inference-time case.

Research Program

The work has three parts.

The first is the scorer. For each item in HLE, GPQA Diamond, and SimpleQA, we decompose the reference rationale (or, where none exists, the answer plus annotator-written key steps) into atomic claims. We also mark one "crux" claim per item, the step an expert considers hardest to find. An open-weight judge produces $p(d \models c)$. We validate against human entailment judgments on a stratified sample of several hundred document-claim pairs, using Q-CARE's annotation protocol, and we report agreement separately for verbatim and paraphrased leaks.

The second is a controlled injection study, which provides ground truth. We wrap an open-weight reasoning model in a standard search-and-read loop over a fixed local index: a web snapshot from which all known benchmark mirrors have been removed. For each item, we inject one synthetic page at a controlled dose. The page reveals zero, one-third, two-thirds, or all of the solution claims, with and without the crux, in verbatim, paraphrased, and fragmented-across-pages variants. Comparing accuracy against the same item with no injected page gives causal inflation as a function of dose. This also answers the open question raised by the STC work: when a page covers part of the solution, does the agent finish the rest correctly, or does it confabulate the remainder?

The third is an observational audit. We run commercial search agents, including the Sonar models audited in the STC paper, on the same benchmarks and log every retrieved page. We score each page with $\ell$, then estimate how much of each agent's accuracy is attributable to leakage at each dose. We use the dose-response curves from the injection study to adjust for confounding by item difficulty, and we compare against each agent's closed-book accuracy on the same items.

Baselines

The naive reference point is no audit at all: report the agent's raw benchmark accuracy as its capability. Every other method is judged by how much of the gap between raw accuracy and leakage-free accuracy it explains. In the injection study, leakage-free accuracy is the no-injection condition. In the observational study, it is the accuracy of the same agent on a mirror-free index.

The strongest existing method is the URL-matching audit of Search-Time Data Contamination. It flags an item when any retrieved URL resolves to a known benchmark mirror; its headline result is 37.5% versus 17.4% for Sonar Deep Research on flagged versus unflagged HLE items. We rerun it on exactly the same retrieval logs as our scorer, with the same mirror list. It beats us if the accuracy gap between its flagged and unflagged items accounts for as much of the total inflation as our graded score does. We count our method as better only if two things hold. Items it calls clean but our score assigns substantial dose must show statistically significant inflation over closed-book accuracy. And our score must explain more variance in per-item "search turned a wrong answer into a right one" outcomes, measured by AUROC, on held-out items.

A stronger content-based binary baseline is exact or normalized answer-string matching: flag the item if the gold final answer appears in any retrieved page. A cheap continuous baseline is n-gram overlap between each page and the reference rationale. We found no reported result on Lacuna for either as a search-time contamination detector, so we establish their numbers ourselves on the same logs.

Two ablations isolate the key ingredient. The first replaces claim decomposition with whole-document embedding similarity to the reference solution. It costs the same to run per page but has no decomposition. If it matches our AUROC and dose-response fit, decomposition is unnecessary. The second replaces direct probability readout with a judge that reasons before its verdict, which tests whether the ranking-resolution loss reported in the Beyond Polarization paper carries over to leakage scoring. All methods use the same judge backbone where one is needed, the same logs, and the same threshold-tuning split. Thresholds for binary methods are tuned to maximize their own AUROC, so they are not handicapped.

Why not just…?

Why not just match the gold answer string in retrieved pages? For short-answer benchmarks this is a serious threat, and we expect it to work nearly as well as our score on SimpleQA, where the answer is a single fact. That is why SimpleQA serves as our control. On HLE and GPQA, the final answer is often a number or a multiple-choice letter that is either absent from leaking pages or present by coincidence. The leak is instead an intermediate result. The injection study settles the question directly: if partial-dose pages without the final answer string produce no inflation, string matching suffices and we will say so.

Why not just block benchmark domains, or rely on the URL audit? The STC authors themselves report that third parties re-host datasets and strip canary strings, and that repeated evaluation drives community solutions into search indexes. Blocking known hosts cannot catch a forum post that paraphrases a solution. The observational audit measures how much inflation remains on items that pass URL filtering. If that residual is negligible, our contribution narrows to a validation of the existing audit, and that is a publishable negative result.

Why not just use embedding similarity, or n-gram overlap, to the reference solution? These are continuous and cheap. Embedding similarity is our main ablation precisely because it could plausibly work as well as decomposition. Our argument against it is specific and testable. A page discussing the same topic without revealing the solution will look similar, while a page giving only the crux step in different words may not. Given the Test of Time finding that paraphrase defeats surface attribution, we expect n-gram overlap to fail on paraphrased injections. If embedding similarity matches claim coverage on the paraphrase and fragment conditions, we will recommend it instead.

Why not just build private or post-cutoff benchmarks? Fresh benchmarks help with training-data leakage, but they do not audit the many results already reported on public sets. As the STC paper notes, temporal cutoffs offer no protection to agents that browse the live web at test time; a private benchmark is contaminated the first time someone posts a solution.

Proposed Experiments and Decisive Checks

The primary experiment is the injection study across the three benchmarks. It produces accuracy-versus-dose curves for each leak style, and it separates proportional dose effects from crux effects by comparing equal-coverage pages with and without the crux claim. Two secondary measurements come from the same runs: confabulation rate, defined as wrong completions that are consistent with the leaked partial steps, and judge agreement with humans on paraphrased leaks.

Three outcomes would be decisive. The idea is supported if three conditions hold on HLE: partial-dose injections without the final answer string raise accuracy significantly above the no-injection condition; in the observational audit, items unflagged by both URL and string matching but with high $\ell$ show inflation over closed-book accuracy; and our score's AUROC for predicting search-induced correctness exceeds every baseline's on held-out items. The idea is refuted, and we would report it that way, if inflation appears only when the full answer is present, or if embedding similarity matches claim coverage everywhere. GPQA, which the STC paper found relatively resilient, and SimpleQA, where string matching should suffice, act as contrast cases. A useful score should show low dose and little inflation on the former, and should match the simple baseline on the latter.

As an optional extension, we inject graded leaks into the corpus of BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval (code at xlang-ai/BRIGHT). There we ask whether BM25 (14.5 nDCG@10 reported by BRIGHT) and a rubric-based reasoning reranker in the style of Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval (36.6 nDCG@10 at 7B) rank partial leaks above genuine supporting documents. The answer determines how much leakage reaches the generator in retrieval-augmented pipelines.