Search-enabled language models can now look up the answer key to a benchmark while they are being tested on it. When an agent with web access is asked a question from a public benchmark, its search engine often lands on a dataset mirror that contains that exact question together with its answer. The agent then copies the answer. The only existing audit of this problem catches it by spotting telltale URLs such as huggingface.co/datasets in the agent's search log. That audit fails as soon as the answer key sits on a blog, a stripped JSON mirror, a cached snippet, or behind an API that hides its retrieval log.
This note proposes a black-box test that needs only the agent's final answers. The test exploits the main way the leak happens: verbatim benchmark wording is an excellent search query for the page that holds the answer key. We rewrite each benchmark question at controlled distances from its original wording, keeping the named entities that genuine retrieval needs and varying only the words that could be swapped for synonyms. We also build "counterfactual twins", minimal edits that keep most of the original wording but change the correct answer. An agent that reasons from evidence should be roughly as accurate on light rewrites as on the original, and it should give the new answer on the twin. An agent that is copying from an answer key should lose accuracy sharply on the first small rewrite. On the twin it should repeat the original benchmark answer, a mistake we call an echo, which no honest reasoner has a reason to make.
Core Claim
Search-time contamination leaves a behavioral fingerprint that can be read without URLs, retrieval logs, or corpus access. The fingerprint has two parts. The first is a sharp accuracy drop at small lexical distance from the canonical wording; this drop is absent when the same agent runs closed-book or with search limited to clean sources. The second is a high echo rate on counterfactual twins. Taken together, these two signals give a per-item and per-system contamination estimate. We claim this estimate recovers planted contamination far better than URL matching when mirrors are obfuscated. When full retrieval logs are available, we expect it to roughly match snippet-overlap checks, and our claim is only that it remains usable when logs are not.
Where This Sits in Prior Work
Search-Time Data Contamination motivates the proposal and defines the setting. It shows that Perplexity Sonar agents retrieve HuggingFace copies of Humanity's Last Exam (HLE), SimpleQA, and GPQA, and that about 3% of samples are contaminated through such links. Sonar Deep Research scores 37.5% on contaminated HLE items against 17.4% on uncontaminated ones, and some models reach 100% on contaminated SimpleQA items. That paper detects contamination with URL and acronym matching plus manual reading of reasoning chains. We keep its threat model and replace the detector with one that does not depend on metadata.
Two papers supply the method. The Data Contamination Quiz detects memorization by asking a closed-book model to pick the original wording out of synonym-perturbed variants; it estimates roughly 95% contamination of XSum for GPT-4. We borrow its central insight, that memorized text is tied to its exact surface form. We apply it to a different channel, the search query rather than the weights, and we test it through task accuracy rather than a recognition quiz. Lexical Diversity-aware Relevance Assessment for Retrieval-Augmented Generation (DRAG) splits queries into invariant parts, meaning entities that must match exactly, and variant parts, meaning predicates with many synonyms, so that retrieval becomes robust to rewording. We use the same decomposition for the opposite purpose. Holding the invariant parts fixed preserves legitimate retrieval, and perturbing only the variant parts breaks the verbatim hook.
Counterfactual Reasoning for Retrieval-Augmented Generation (CF-RAG) both supplies a method and constrains the design. It generates counterfactual queries in order to penalize passages that support both the original and the altered question. The research file notes that a full benchmark dump supports every question. CF-RAG's relevance scoring therefore breaks down on contaminated sources. We turn that breakdown into a detection signal: if an agent's answer to the twin matches the original answer, the evidence it used was indifferent to the edit, which is exactly what an answer key looks like.
Two papers constrain how we build controls. Estimating Contamination via Perplexity shows that reading-comprehension and summarization sets such as QuAC, SQuAD v2, and XSum are heavily memorized in the weights. Proving Test Set Contamination in Black-Box Language Models shows that ordering memorization appears after only 2–4 exposures. Because of these results, any accuracy drop under paraphrase could come from pretraining memorization rather than search. We therefore always measure the drop against the same model run closed-book.
Finally, Lexical Hints of Accuracy in LLM Reasoning Chains finds that uncertainty words in reasoning traces predict failure much better than self-reported confidence. The Matthews correlation coefficient (MCC, a correlation measure for binary predictions) is 0.215 versus 0.085 on HLE. We treat reasoning-trace lexical statistics, including length-robust diversity measures such as the moving-average type-token ratio (MATTR, the share of distinct words averaged over sliding windows), only as a secondary signal. We test whether these statistics add anything once the wording probes are in place, and we do not assume they do.
Research Program
The first component is a probe generator. For each benchmark item, an auxiliary model labels invariant and variant spans using the DRAG decomposition. It then produces rewrites at graded lexical distance $d$, measured as the share of variant-span tokens changed and confirmed by n-gram overlap with the original. Invariant spans are never touched. Human spot-checks and an entailment filter keep only meaning-preserving rewrites.
Where the domain allows it, the generator also writes a counterfactual twin, $d$ stays small but one answer-determining detail changes. In SimpleQA and PopQA this means a date, a role, or an attribute; in math-style HLE items it means a numeric value. Each twin's new gold answer is verified independently. We do not build twins for HLE items where verification is infeasible, and for those items the paraphrase arm is used alone.
The second component is the estimator. Let $a_{s}(d)$ be the accuracy of a system at distance $d$ with search on, and $a_{0}(d)$ the accuracy of the same model closed-book or with search restricted to a clean index. The search-specific wording dependence is
$$\Delta = \big[a_{s}(0) - a_{s}(d^{*})\big] - \big[a_{0}(0) - a_{0}(d^{*})\big],$$
where $d^{*}$ is the smallest distance that consistently breaks verbatim retrieval. This difference-in-differences cancels two confounds: the extra difficulty a paraphrase adds, and parametric memorization. The second statistic is the echo rate $e$, the fraction of twins on which the system outputs the original item's gold answer. For each item, we combine a large individual accuracy drop with an echo into a contamination flag. The flag is calibrated against planted ground truth.
The third component is that planted ground truth. We build a private search index from a clean web or Wikipedia snapshot and insert benchmark answer keys under four levels of obfuscation: a verbatim HuggingFace-style JSON file, the same file with URL and canary strings removed, a reformatted blog-post mirror, and a paraphrased answer-key page. For each item we control whether a key is planted, so every item has a known contamination label.
Baselines
The naive reference point is to do nothing. We report raw accuracy with search on and treat any gain over closed-book as legitimate. This shows how much contamination inflates headline scores when no audit is run.
The strongest existing method is URL and keyword matching from Search-Time Data Contamination. It flags an item if any retrieved URL points to a dataset host or names the benchmark. The best number in the file attached to this method is the 37.5% versus 17.4% HLE accuracy gap between flagged and unflagged items for Sonar Deep Research. No reported result was found on Lacuna for the precision or recall of URL matching against ground-truth contamination. We run it on the same planted index, on the same items, with full access to retrieval logs, which favors this baseline. Beating it means higher recall at equal precision on the stripped-URL, blog-mirror, and paraphrased-mirror conditions, while staying within a few points on the verbatim-JSON condition.
The second baseline is snippet overlap. With full logs, we flag an item if any retrieved text contains a long n-gram match to the question together with the gold answer string. Where logs are available, this is the real competitor, and we expect it to be close to perfect on verbatim mirrors. To keep the comparison fair, we use the same thresholds-tuning budget on a held-out split. Our method counts as useful against this baseline if it matches snippet overlap within a small margin while using no logs, and if it degrades less on the paraphrased-mirror condition, where n-gram matching should weaken.
Two ablations isolate the key ingredients. A single-level paraphrase ablation compares one fixed rewrite, without graded distance, without a closed-book control, and without twins. It tests whether the dose-response curve and the difference-in-differences actually add anything. An unconstrained-paraphrase ablation lets rewrites alter invariant entities. It tests whether keeping entities fixed is what separates broken retrieval hooks from broken legitimate retrieval. The reasoning-trace signal, harmful-word counts as in Lexical Hints of Accuracy plus MATTR, is tested both as a standalone detector and as an addition to our flag.
Why not just…?
Why not just block dataset domains and rely on canary strings? The search-time contamination paper itself reports that canaries are stripped when datasets are mirrored, and that repeated evaluation traffic causes search engines to cache questions and answers as ordinary snippets. In our planted index, URL blocking will remove the verbatim-JSON condition by construction. The other three conditions directly measure how much contamination survives blocking, and if very little does, the proposal loses much of its motivation.
Why not just check retrieved snippets for n-gram overlap with the answer key? When logs are available, this is a serious threat, and on verbatim mirrors we expect it to tie our method. The case for wording probes rests on two settings. The first is a commercial system that exposes only citations or nothing at all. The second is a paraphrased mirror, where surface overlap drops but, as the DCQ results suggest for memorized text, the answer is still clearly recoverable. If snippet overlap also handles paraphrased mirrors well, we will report that our contribution is limited to the black-box setting.
Why not just evaluate on fresh dynamic benchmarks like LiveBench? Fresh benchmarks stop future leakage, but they cannot audit the large body of scores already reported on HLE, SimpleQA, and GPQA, and they leak in turn once they are published. Wording probes can be applied after the fact to any existing benchmark and any deployed agent.
Why not just paraphrase the whole benchmark once and re-score? A single paraphrase mixes up three effects: the added difficulty of the rewrite, pretraining memorization of the original wording, and search-time copying. The single-level ablation measures how badly that confusion hurts. We expect the closed-book difference and the echo test to be what separates search-time contamination from the other two.
Proposed Experiments
The first experiment uses the planted index with an open-weight agent built on Llama3-8B-Instruct, the model DRAG uses, on SimpleQA, PopQA, and HotpotQA. Every item has a known label across the four obfuscation levels. We report precision and recall per level for our flag and for each baseline, together with the $\Delta$ curves across distance. The decisive check is whether recall on the stripped and paraphrased conditions clearly exceeds URL matching, and whether false positives stay low on unplanted items in the HotpotQA set, where HotpotQA's own Wikipedia sources are legitimately retrievable.
The second experiment audits deployed search agents, Perplexity Sonar variants as in the search-time contamination paper, on HLE, SimpleQA, and GPQA. The items that paper flagged by URL serve as partial labels. We test whether our flag catches those items, and how many additional, unlinked items it flags. GPQA, which that paper describes as relatively resilient to search, serves as a negative control and should show a small $\Delta$ and a near-zero echo rate.
A sanity check on parametric memorization runs the closed-book arm on XSum-derived and SQuAD v2-derived question sets, where pretraining memorization is documented. We expect a clear closed-book drop under paraphrase, and the difference-in-differences should remove it. This confirms that $\Delta$ isolates search rather than weights.
The final check asks whether reasoning-trace lexical statistics, including MATTR shifts at the point where an agent starts extracting from a retrieved source, add measurable MCC on top of the wording flag. A null result here would be reported as such.
References
- Search-Time Data Contamination
- Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
- Lexical Diversity-aware Relevance Assessment for Retrieval-Augmented Generation
- Counterfactual Reasoning for Retrieval-Augmented Generation
- Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation
- Proving Test Set Contamination in Black-Box Language Models
- Lexical Hints of Accuracy in LLM Reasoning Chains
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark