When a language model can search the web during evaluation, it can sometimes find the benchmark's answer key. Han et al. showed this happens in deployed systems: Search-Time Data Contamination reports that Sonar Deep Research scored 37.5% on Humanity's Last Exam questions whose answers were findable on HuggingFace mirrors, against 17.4% on the rest. Today the only way to catch this is to know where the leak lives and match it by URL or string. We propose a check that does not need to know where the leak is. Each benchmark question is paired with a "twin": a minimally edited version whose correct answer is different but for which honest supporting evidence is equally available. Honest retrieval should help the original and its twin about equally. A leaked answer key helps only the original, because the key contains the original's answer and nothing about the twin. We measure the extra help the original gets when search is on, relative to a reference condition without leaks. That excess estimates how much of the score comes from contamination, whatever form the leak takes: verbatim, templated, or paraphrased.
We develop and validate this check inside hybrid retrieval, meaning systems that combine keyword matching such as BM25 with dense embedding search. In that setting we can plant leaks ourselves, so the true inflation is known. This also lets us test a hypothesis that links the three directions in this proposal. The features that make hybrid retrievers and noise-robust readers good at rejecting misleading "distractor" passages are exact entity and question-string matching, and a preference for evidence that agrees with the query. Those same features should make them better at surfacing and trusting leaked answer keys. If that is true, making retrieval more robust to noise quietly makes contamination worse.
Core Claim
We make two claims. First, a difference-in-differences estimate over original/twin pairs recovers search-time contamination inflation to within a few accuracy points. It should do so where URL checkers and string-overlap decontamination fail, namely on leaks hosted off HuggingFace and on paraphrased or templated leaks. Second, leak exposure in a hybrid retriever rises with the weight placed on the sparse keyword component. Defenses against contextual noise, specifically discriminator filtering and counterfactual arbitration, should reduce distractor damage while leaving leak-driven inflation unchanged or larger. The estimator is the main contribution. The trade-off is the scientific finding that justifies it: benchmark scores from noise-robust RAG systems cannot be taken at face value.
Relation to Prior Work
Three papers supply the motivation. Search-Time Data Contamination documents the phenomenon in commercial search agents and detects it with a HuggingFace contamination checker. It treats the retriever as a black box and cannot see leaks on other hosts. Retrieval Collapses When AI Pollutes the Web compares BM25 and an LLM ranker under synthetic pollution. It finds BM25 lets 19–24% adversarial exposure through at 67% pool contamination, while the LLM ranker holds near zero. It never tests fused sparse-dense retrievers and never looks at benchmark answer keys. SoftMatcha 2 shows that exact matching misses "template leakage": it finds 36 additional contaminated MMLU/GSM8K/MATH items at 81% manual precision. That is the reason we plant templated leaks. SoftMatcha 2 is an offline audit that requires the benchmark text and index access, and our estimator requires neither.
Two papers supply the defenses we test for side effects. Why So Gullible? trains a discriminator that flags entity-swapped "counterfactual noise" passages in Fusion-in-Decoder, a reader that encodes each retrieved passage separately and fuses them when generating. Counterfactual Reasoning for RAG penalizes documents that support contrasting perturbed queries equally. Our twin questions resemble that method's counterfactual queries. The difference is purpose: that work uses counterfactual queries to rank documents, and we use twins to measure a benchmark score. Both defenses reward passages that agree tightly with the query, and a leaked key does exactly that. That is the basis of our hypothesis that they amplify contamination.
Two papers supply the retrieval machinery. Sparse, Dense, and Attentional Representations for Text Retrieval provides linear score interpolation and ME-HYBRID, which combines multi-vector document encodings with BM25. DyVal generates items procedurally to avoid contamination and informs how we perturb math items. DyVal constructs whole new benchmarks. We instead keep the existing benchmark and use perturbation only to calibrate it. Finally, CoDeC and On The Fragility of Benchmark Contamination Detection in Reasoning Models constrain our design. Probability-based detectors target contamination stored in the model's weights. They can be cloaked by RL post-training, which drops them to 50–58% AUROC, and they say nothing about answers found at inference time. We therefore use accuracy differences rather than model internals.
Research Program
The estimator works as follows. For a system $S$, let $A_S(o)$ and $A_S(t)$ be accuracy on original items and on their twins. Let $R$ be a reference condition with no search-time leaks. In the controlled study, $R$ is the same retriever over a corpus without planted leaks. For black-box agents, $R$ is the model run closed-book, which absorbs contamination already stored in the weights. The inflation estimate is
$$\hat{I} = \big[A_S(o) - A_S(t)\big] - \big[A_R(o) - A_R(t)\big],$$
The second bracket removes any intrinsic difficulty gap between originals and their twins. Validity rests on one assumption: honest evidence helps originals and twins equally. We enforce this by construction.
We build twins for three benchmark types. For SimpleQA and NQ-open, a twin swaps the queried entity for a sibling entity of matched Wikipedia popularity whose answer is present in the corpus in a comparable passage. For GSM8K, a twin changes the numbers, following DyVal-style perturbation, so the reasoning is identical and the answer differs. For a subset of HLE, a twin keeps the question's structure and changes a parameter, reusing only items where an expert can verify the new answer.
For the controlled corpus we use Wikipedia for NQ/SimpleQA and a FineWeb-Edu slice (The FineWeb Datasets) for GSM8K. We plant leaks in three forms. A verbatim form is a dump file listing question and answer. A templated form is a leak with changed names or numbers but the original's answer structure. A paraphrased form is a forum-style post discussing the question. Leaks are hosted under HuggingFace-like, GitHub-like, and generic blog-like URLs. Alongside them we plant counterfactual distractors in the style of Why So Gullible. Leak rate and noise rate are varied independently.
Retrievers are BM25, a dense bi-encoder, a multi-vector encoder, and interpolated hybrids that sweep the sparse weight $\alpha$ from 0 to 1. Readers are a Fusion-in-Decoder model and an instruction-tuned open LLM. Each reader is tested plain, with the discriminator, and with counterfactual arbitration. Because we control where the leaks are, the true inflation is simply accuracy with leaks minus accuracy with the leaks deleted. Every estimator is scored against that number.
Baselines
The naive reference point is reporting search-augmented accuracy as is. It implicitly assumes zero inflation, so its error equals the true inflation. Any method must beat this by a wide margin to be worth anything.
The first detection baseline is the URL-based contamination checker from Search-Time Data Contamination. It flags an item as contaminated when a retrieved source is a HuggingFace benchmark mirror, and inflation is estimated as the accuracy gap between flagged and unflagged items. This is the method actually used for the phenomenon, and it is the strongest competitor for verbatim leaks on known hosts. We run it on identical retrieval logs. We expect it to match our estimator on HuggingFace-hosted verbatim leaks and fail on the other hosts and forms. The file reports its headline gap (37.5% versus 17.4% on HLE for Sonar Deep Research) but no accuracy figure for detecting leaks. We will measure that ourselves.
The second baseline is index-side decontamination. We run n-gram overlap and SoftMatcha 2 soft matching of the benchmark text against the corpus, and estimate inflation by recomputing accuracy with matched documents removed. This is the strongest method when the evaluator can access the index. It reports 36 extra template leaks at 81% precision on MMLU/GSM8K/MATH (SoftMatcha 2). It gets the same corpus and the same match-threshold tuning budget, tuned on a held-out leak split. The twin estimator beats it only if it has lower absolute error against ground truth on paraphrased leaks. On verbatim and templated leaks we expect a tie.
On the retrieval side, the strongest reported hybrid is ME-HYBRID at MRR@10 0.343 on MS MARCO passages, versus 0.302 for a single-vector dual encoder (Sparse, Dense, and Attentional Representations). We use it as the high-quality retrieval point on the $\alpha$ sweep and confirm that our reimplementation roughly reproduces the gain on MS MARCO before planting leaks. The strongest noise defense we test is the Why So Gullible discriminator, which reports +5.8 EM on NQ-open with 35% noise. We retrain it on our distractors with its original recipe.
Two ablations isolate the estimator's key ingredients. The first drops the reference correction and uses the raw original-minus-twin gap. This tests whether the difference-in-differences is needed. The second uses twins without matched honest evidence in the corpus. This tests whether evidence parity is the load-bearing assumption. We count as success a mean absolute error below 2 accuracy points against true inflation, pooled across leak forms and hosts, with no single form above 4. Both baselines must show at least double that error on non-HuggingFace or paraphrased leaks.
Why not just…?
Why not just block HuggingFace and GitHub domains at search time? For verbatim dumps on known hosts this works, and our URL-checker baseline will show it. The problem is that leaks migrate to blogs, forums, and mirrors, and blocking certifies nothing about what remains. The host-by-form grid measures exactly how much inflation a blocklist leaves behind. If it is negligible across all non-HuggingFace hosts in our corpus, we will say so and narrow the claim to paraphrased leaks.
Why not just decontaminate the retrieval index against the benchmark text? This is the real threat. When the index is accessible and leaks share surface form with the benchmark, soft matching may catch nearly everything. The paraphrase condition decides the matter. If SoftMatcha-style matching recovers paraphrased-leak inflation as accurately as twins do, the controlled-corpus advantage vanishes. The estimator would then be useful only for black-box commercial search, where no index is available. We would report that outcome as the finding.
Why not just evaluate against a corpus snapshot frozen before the benchmark's release? This is a sound control, and in the controlled study it is our reference condition. It cannot evaluate live search agents, which is where the observed contamination happens, and it penalizes legitimate access to newer information. The twin estimator is meant for the case where the snapshot is unavailable. We validate it precisely by checking that it agrees with snapshot-based ground truth.
Why not just prompt the model to ignore answer keys? The evidence argues against this. Why So Gullible found GPT-3.5 did worse with noisy retrieval than without it, and the search-time study found reasoning models abandoning their own derivations to copy retrieved labels. We include an "ignore leaked answers" instruction as a reader variant so the claim is tested rather than assumed.
Proposed Experiments
The first decisive check is estimator accuracy. We compute $\hat{I}$ against true inflation across the full grid of leak form, host, leak rate, noise rate, and retriever, and compare its error with the URL checker and index decontamination. The second check is the trade-off. We plot true inflation and distractor-induced accuracy loss against $\alpha$, and against defense on or off. The hypothesis fails if higher sparse weight or the discriminator does not raise leak exposure in the top-10 results or raise inflation. The third check is transfer. We apply $\hat{I}$, with a closed-book reference, to a black-box search agent on SimpleQA and the HLE twin subset. We then test whether the items it implicates concentrate where the URL checker finds leaks, and whether it flags additional inflation where the checker finds nothing. A final sanity check guards against false positives. On a corpus with no leaks at all, $\hat{I}$ must be indistinguishable from zero. Otherwise twin construction is biased and the estimator is not usable.
References
- Search-Time Data Contamination
- Retrieval Collapses When AI Pollutes the Web
- SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora
- Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise
- Counterfactual Reasoning for Retrieval-Augmented Generation
- Sparse, Dense, and Attentional Representations for Text Retrieval
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Detecting Data Contamination in LLMs via In-Context Learning
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale