When a language model that can search the web is graded on a public benchmark, it can simply find the answer key. Third-party copies of benchmarks such as Humanity's Last Exam sit on HuggingFace and GitHub, and the model copies the answer instead of solving the problem. Search-Time Data Contamination documented this "search-time contamination." It is tempting to think that modern retrieval already offers a partial defense. Many systems now remove redundant results, meaning near-duplicate pages, so it seems natural that the many duplicate mirrors of an answer key would be thinned out. We argue the opposite. Removing redundancy reduces how many copies of the key reach the model, but the model needs only one. Diversity-seeking selection may also actively promote a lone, unusual-looking mirror page into the context. If that holds, deduplication leaves a floor of contamination that it can never remove, and on single-copy items it may raise exposure.
This note proposes a controlled study of that claim and a defense built from the same selection machinery. The defense penalizes retrieved pages for repeating the benchmark item itself, rather than for repeating each other. It also proposes a cleaner way to measure how much a leak inflates scores. Instead of comparing different questions, as prior work does, we compare the same question answered with and without its leaked pages.
Core Claim
The claim has three parts. First, a model's exposure to a leaked answer key depends on whether at least one copy enters its context, not on how many copies there are. Redundancy-aware selectors leave that exposure largely intact. These are AdaGReS, which picks passages under a token budget while penalizing overlap (AdaGReS), fixed-weight Maximal Marginal Relevance (MMR), and diversity-aware graph search (Graph-Based Algorithms for Diverse Similarity Search). Because these selectors free up context slots, they can also pull a single low-ranked mirror into the context.
Second, the lever that does work points redundancy at a different target. A leak page is unusually redundant with the test question: it repeats the question's clauses, its answer options, and its specific numbers. Honest evidence tends to support the answer without restating the exam item. We therefore add an "item-echo" penalty to a set-level selection objective. The penalty is computed over the item's own content units, in the spirit of the nugget decomposition used by CRUX. CRUX rewards coverage of answer nuggets; we penalize coverage of question nuggets.
Third, the true size of the inflation must be estimated per item by counterfactual removal of the leaked pages. The headline comparison in Search-Time Data Contamination, 37.5% accuracy on contaminated versus 17.4% on uncontaminated HLE items for Sonar Deep Research, compares different questions. Questions that get mirrored and discussed may also be easier, so that comparison mixes leakage with item difficulty.
Where This Sits
The work is motivated by two papers. Search-Time Data Contamination shows that agents retrieve answer keys and abandon their own reasoning to use them. It detects leaks with keyword and domain checks and explicitly leaves open how ranking and deduplication affect exposure. Retrieval Collapses When AI Pollutes the Web simulates SEO-style synthetic pages flooding MS MARCO results. At 67% pool contamination, more than 80% of top-ranked exposure was synthetic for both BM25 and an LLM ranker. An LLM ranker suppressed adversarially altered content but not highly relevant synthetic content. Answer keys are the extreme case of highly relevant content, so we expect LLM rerankers to promote them. That paper calls for evidence diversity but never tests a diversification algorithm. We do, and we predict that diversity alone is insufficient.
The method comes from the redundancy literature. We reuse AdaGReS's set objective, which is approximately submodular and has an adaptive redundancy weight, so greedy selection carries a near-optimality guarantee. Submodular here means each added passage helps less as the set grows, which is what makes greedy selection provably close to the best set. We also reuse diverse graph search with web domain as the diversity "color," and CRUX's nugget-coverage metrics to verify that honest evidence coverage is preserved. Retriever Portfolios supplies a further constraint. Mixing retrievers beats retrieving more from a single one (two retrievers fetching four documents each beat one fetching twenty). Diverse portfolios may therefore be especially likely to surface an off-cluster mirror, which makes them a test case for the first claim.
The contamination-detection work constrains what a retrieval-side defense must beat. Detecting RLVR Training Data via Structural Convergence of Reasoning detects contaminated prompts from the collapsed diversity of sampled reasoning. Its Min-kNN Distance score reaches AUC 0.70 on average across 1.5B–32B models. AUC is the area under the detection curve, where 0.5 is chance and 1.0 is perfect. A Narrowing Geometry in Contaminated Reasoning finds collapsed internal representations under training-time leakage. Both target leakage stored in the model's weights. A model copying a retrieved key may show similar output collapse, so we treat output-diversity detection as a competing post-hoc detector rather than ignoring it.
Research Program
We build a frozen, reproducible search environment rather than querying the live web, which keeps changing. Following the setup in Retrieval Collapses When AI Pollutes the Web, which drew a 10,000-document pool from a search API, we snapshot organic search results for every item in HLE, SimpleQA, and GPQA. We remove any leak pages already present, using a strict combination of domain, canary-string, n-gram, and manual checks. We then plant synthetic mirror pages under controlled conditions.
Two factors vary. The first is multiplicity $m \in \{0,1,2,4,8,16\}$ copies of an item's leak, spread across distinct synthetic domains. The second is the form of the leak. We use five forms: verbatim dataset rows, reformatted rows (JSON rendered as an HTML table or forum post), LLM-paraphrased question-plus-answer pages, translated pages, and "answer-only" pages that list item identifiers and answer letters.
We also take the real leak pages we initially removed and re-insert them as a natural stratum. This lets us check that the synthetic findings transfer to real mirrors.
The selection objective extends AdaGReS. For a benchmark item $x$ and a candidate context set $C$, we select $C$ to maximize
$$F(x, C) = \alpha \,\mathrm{Rel}(x, C) - \beta \,\mathrm{Red}(C) - \gamma \sum_{d \in C} \mathrm{Echo}(x, d).$$
Here $\mathrm{Rel}$ and $\mathrm{Red}$ are AdaGReS's relevance and inter-passage redundancy terms, with $\alpha$ and $\beta$ set by its closed-form calibration. $\mathrm{Echo}(x, d) \in [0,1]$ is the fraction of item $x$'s content units that document $d$ reproduces, judged by semantic matching rather than exact string overlap. The content units are question clauses, option strings, numeric constants, and the benchmark name or ID. $\gamma$ is the echo weight. Because the echo term is a modular sum, a sum of independent per-document scores, adding it keeps the approximate-submodularity argument intact, and greedy selection still applies. A hard-threshold variant drops any document with $\mathrm{Echo}$ above $\tau$. Comparing soft and hard variants tells us whether a simple filter is enough.
Answer-only pages are the weak point of this design. They reproduce few question units. We expect them to be caught mainly by the ID and benchmark-name units, and we report this stratum separately and honestly.
Baselines
The naive reference point is plain pointwise top-k retrieval: BM25, a dense retriever, and an LLM reranker, each feeding the same answer models. We use open models (Qwen2.5-7B and 32B, Llama3.1-8B) so that every condition is rerun deterministically on the same frozen corpus with identical prompts and token budgets.
The redundancy baselines are fixed-weight MMR, AdaGReS without the echo term, diverse graph search with domain as color (Graph-Based Algorithms for Diverse Similarity Search), and a two-retriever portfolio in the style of Retriever Portfolios. For reference, AdaGReS reports an 8–15 percentage-point IOU gain over top-k on a high-redundancy biomedical corpus. These baselines exist to test the first claim, not to be beaten on leakage, since they were never designed for it.
The strongest leakage competitors are two cheap filters applied to the retrieved set. The first is the HuggingFace and GitHub domain checker plus canary-string matching used by Search-Time Data Contamination. The second is a 13-gram overlap filter against the item's own question and option text. The 13-gram filter is the real threat to this proposal. We give it the same knowledge of the benchmark that the echo penalty has, and we tune its threshold on the same held-out split and budget used to tune $\gamma$ and $\tau$.
No reported result was found on Lacuna for any retrieval-side defense against search-time contamination. The comparison numbers available are the exposure-driven gaps reported by Han et al.: 37.5% versus 17.4% on HLE, and 100% on contaminated SimpleQA items. The output-diversity detector Min-kNN Distance (AUC 0.70, Detecting RLVR Training Data via Structural Convergence of Reasoning) serves as the post-hoc detection baseline, run with its specified 32 samples per item.
The key ablation swaps semantic echo for exact n-gram echo inside the same objective. This isolates whether semantic matching matters, separately from the set-selection framing. Beating the baselines means two things. Leaked-page exposure must fall to near the $m=0$ level on the paraphrased and reformatted strata, where we expect n-gram filtering to fail. And CRUX coverage plus answer accuracy on clean items must stay within noise of AdaGReS without echo.
Why not just…?
Why not just block HuggingFace and GitHub and match canary strings? Han et al. note that canary strings are often stripped when datasets are re-hosted. Their own checker targets specific domains, while mirrors and discussions spread to arbitrary sites. Our planted mirrors on synthetic domains measure exactly how much a domain blocklist misses. We expect it to catch the verbatim stratum on known domains and almost nothing else.
Why not just n-gram-match retrieved pages against the benchmark item? This is the serious alternative. On verbatim and lightly reformatted mirrors it will plausibly match the echo penalty at a fraction of the cost. The proposal does not depend on beating it there. The decisive cells are the paraphrased, translated, and answer-only strata. If n-gram filtering matches semantic echo on those too, we will report that the cheap filter suffices. The surviving contribution would then be the one-copy-floor finding and the counterfactual inflation protocol, which no filter comparison can supply.
Why not just retrieve more documents, or trust the model to ignore answer keys? Retrieving more only raises the chance that some copy gets in. Han et al. show models do not ignore keys: reasoning traces explicitly discard the model's own calculation once the label is found. Our multiplicity sweep tests both points directly by varying $k$ alongside $m$.
Why not just detect leakage afterwards from collapsed output diversity? Min-kNN Distance was built for leakage stored in model weights and costs 6.65 seconds per item for 32 samples. It also flags an item only after the score has been earned, so it audits rather than prevents. We run it as a detector on our conditions. If it reliably flags retrieval-driven copying, it complements a retrieval-side defense rather than replacing it.
Why not just switch to dynamically generated benchmarks? DARG perturbs reasoning graphs for math and logic. It does not cover knowledge-intensive questions like HLE or SimpleQA, where consulting the web is the point. Han et al. also observe that repeated evaluation itself gets crawled and indexed, so new items leak too.
Proposed Experiments
The first decisive check is the exposure curve. For each selector, we plot the probability that at least one leak page enters the context, and the resulting accuracy, against multiplicity $m$. The claim predicts three patterns: pointwise top-k exposure rises with $m$; redundancy-aware selectors flatten the leak count but keep exposure high once $m \ge 1$; and at $m=1$, diversity-seeking selectors expose more than top-k does. If redundancy removal instead drives exposure toward zero at high $m$, the first claim is wrong. In that case the paper becomes a positive result for existing deduplication.
The second check is the counterfactual inflation estimate. For each item and selector, we compare accuracy with the item's leak pages present against accuracy with them removed from the corpus. We set this same-item estimate against the between-item contrast Han et al. use, on the re-inserted real-leak stratum. A large gap between the two estimates would show that item difficulty confounds the published comparison.
The third check is the defense table. It reports leak exposure, accuracy, and inflation for the echo objective, its n-gram ablation, the hard filter, the domain and canary checker, and Min-kNN, broken out by the five leak forms.
The fourth check measures the defense's cost on honest retrieval. We run the echo objective on HotpotQA, Natural Questions, and CRUX-DUC and CRUX-Multi-News, where no leaks exist. We report exact match and CRUX coverage, and α-nDCG, a ranking metric that rewards new information and penalizes repeats. This shows whether penalizing question echo removes legitimate evidence that happens to restate the question. We expect this risk to be highest for SimpleQA-style factoid questions, and we inspect those false positives by hand.
References
- Search-Time Data Contamination
- AdaGReS:Adaptive Greedy Context Selection via Redundancy-Aware Scoring for Token-Budgeted RAG
- Graph-Based Algorithms for Diverse Similarity Search
- Controlled Retrieval-augmented Context Evaluation for Long-form RAG
- Retrieval Collapses When AI Pollutes the Web
- Retriever Portfolios: A Principled Approach to Adaptive RAG
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- A Narrowing Geometry in Contaminated Reasoning
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph