Leak Filters That Fit the Benchmark: Attested Search Proxies Against Search-Time Contamination

Web-searching AI agents can cheat on benchmarks without anyone intending it. They search for help with a question, land on a public copy of the benchmark on HuggingFace or GitHub, read the answer key, and copy it. Search-Time Data Contamination measured this directly. About 3% of questions on Humanity's Last Exam (HLE), SimpleQA and GPQA were exposed through HuggingFace copies, and Sonar Deep Research's HLE accuracy rose from 17.4% on clean items to 37.5% on exposed ones. Hiding the benchmark does not fix this, because copies spread and the marker strings meant to flag them get stripped. Taking search away does not fix it either, because search is the capability being measured.

We propose a search proxy that sits between the agent and the web and removes pages that leak a benchmark item. It lets through pages that only contain facts the agent is legitimately supposed to find. The central observation is that what counts as a leak depends on the benchmark. On HLE, any page containing the question is a leak. On SimpleQA, the answer is a real fact that appears on Wikipedia, so blocking the answer string would break the task; the leak is the question paired with its answer. We therefore choose the proxy's matching rule per benchmark, by automated search, the same way earlier work tuned text features per task. The proxy runs inside a trusted execution environment, meaning a hardware-isolated enclave that can cryptographically prove which code it ran. That lets a model developer run the evaluation on their own machines while proving to others that every retrieval passed through the official filter, and without ever seeing the answer key the filter holds.

Core Claim

A leak filter that inspects page content, with its matching representation tuned per benchmark, can remove most search-time score inflation while leaving accuracy on clean items unchanged. A single fixed rule cannot do both across benchmarks. Blocking by domain, or matching one generic string, either misses reformatted copies or blocks legitimate evidence, and which failure occurs depends on the benchmark. Placing only the proxy in an enclave, rather than the model, makes this checkable by third parties at modest cost.

The evaluation premise is straightforward. With the filter on, accuracy on previously contaminated items should fall to the level of matched clean items. Accuracy on clean items should not move.

Why Representation Selection Is the Right Lens

The representation literature supplies the core intuition. What Makes for Good Views for Contrastive Learning? shows that downstream accuracy follows an inverted U as a function of how much information the representation keeps. Keep too little and the signal disappears; keep too much and the model latches onto irrelevant detail. Feature Dropout shows that no single transformation is safe across tasks: removing color is harmless for coarse shape classification and harmful for fine-grained flower classification. Benchmarks of molecular models such as MolRep reach the same conclusion in another field: no single representation wins everywhere.

These papers motivate our claim; none of them concerns contamination. The method we reuse comes from Bayesian Optimization of Text Representations. That work treats choices such as n-gram range, term weighting and stopword removal as hyperparameters and searches over them with a Tree-structured Parzen Estimator. Within 30 trials it made logistic regression competitive with neural models (82.4% on SST), and the best configuration differed across datasets. We apply the same search to leak detection instead of classification.

Research Program

The filter's search space. A candidate leak-detection representation is specified by three choices. The first is the unit being matched: question text alone, answer string alone, or question and answer co-occurring within a window of $w$ tokens. The second is normalization: case folding, canonical formatting of numbers and LaTeX, and machine translation of non-English pages into English. The third is the matching method: word n-gram overlap over a range of lengths, MinHash Jaccard similarity, or sentence-embedding cosine similarity, each with its own threshold.

For each benchmark $b$, we search for the configuration $\theta$ that solves

$$\max_{\theta}\ \mathrm{Recall}_b(\theta) \quad \text{subject to} \quad \mathrm{FBR}_b(\theta) \le 1\%.$$

Here $\mathrm{Recall}_b$ is the fraction of labeled leak pages the filter blocks. $\mathrm{FBR}_b$ is the false-block rate: the fraction of labeled legitimate evidence pages it wrongly blocks. We use TPE with a budget of about 30 trials, matching the budget in Yogatama et al.

Labeled page pools. For each of HLE, SimpleQA and GPQA we build three kinds of pages. Real leak pages come from logging what open agents retrieve when they search with no filter, then labeling hits by exact match plus manual review. Synthetic leak pages are items rewritten the way mirrors typically rewrite them: converted to JSON or CSV, paraphrased, translated, or split so that questions and answers sit in separate sections. Legitimate evidence pages contain the fact or reasoning an item needs but not the item itself. SimpleQA's source pages are the natural example.

We tune the filter on a portion of the leak and legitimate pages, sampled across the three kinds rather than drawn from one kind alone. We report results on held-out pages.

The attested proxy. The filter runs inside an AWS Nitro Enclave. We follow the wrapper pattern of Proof-of-Guardrail in AI Agents. That work places an open guardrail next to a private agent and has the enclave sign a statement that the output passed through the guardrail's exact code. Two things differ in our design. What we place in the enclave is a retrieval filter, not a safety classifier. And we deliberately keep the model outside. That paper reports enclaves costing about 18.5x more than small commodity machines, and enclave memory is limited, so a proxy is feasible where a frontier model is not.

The proxy works as follows. The benchmark organizer releases the encrypted leak index only to an enclave whose code fingerprint matches the published proxy. The enclave terminates encrypted web connections (TLS) itself, so the host machine cannot alter page content in transit. For each retrieval it writes a signed, append-only log entry containing hashes of the URL and content, the filter's verdict, and the proxy's code fingerprint. It also keeps an encrypted copy of every response it allowed through. That copy lets anyone rerun the evaluation against the same web snapshot, which gives the record-and-replay reproducibility that containerized harnesses such as Bencher provide for offline benchmarks. The append-only logging borrows from SciAgent's audit harness, whose logs are protected by software alone; ours are protected by hardware.

Scope. The attestation proves something specific: every retrieval the agent made through the harness's tools was filtered. It cannot prove that no other retrieval happened. If a hosted model performs its own search behind its API, the proxy never sees that traffic, and Sonar-style products fall outside the guarantee. We therefore run agents whose tools are supplied by the harness, such as OpenHands's CodeActAgent with its browser pointed at the proxy. We report hosted search agents only as unfiltered reference points.

P⁴Bench has a related gap. It cannot tell whether the model or an outside solver produced an answer. We do not claim to close that gap. Our attestation covers the retrieval channel, not who generated the answer.

Baselines

No filter. The naive reference point is the agent with open search. The quantity we care about is the contamination gap: accuracy on contaminated items minus accuracy on clean items matched by subject and difficulty. The file's anchor for how large this gap can be is Han et al.'s figures for Sonar Deep Research on HLE: 17.4% on clean items versus 37.5% on contaminated ones. We will measure the gap afresh for our own agents rather than assume it transfers.

Domain blocklist. The cheapest competitor refuses to fetch from huggingface.co, github.com and other known dataset hosts. It targets exactly the mirrors Han et al. identified, so it is the strongest simple competitor.

Fixed question-text matching. This applies one rule to every benchmark: 13-word overlap with the question text, in the spirit of the string matching Han et al. used to detect contaminated URLs. It is the strongest content-level method with an obvious precedent. We found no reported result on Lacuna for the recall or false-block rate of any in-line filter against search-time leaks, so both this baseline and ours will be measured from scratch.

Ablations. The first transfer ablation applies the configuration tuned on one benchmark to the others. This is the test of the task-dependence claim. The second applies a single configuration tuned on all three benchmarks pooled. The third removes normalization, to isolate its contribution.

Fairness conditions. All filters see the same page pools, and every tuned variant gets the same 30-trial search budget. Every agent condition uses the same model, prompts, tool budget and random seeds. Enclave and non-enclave runs use the same filter code.

What counts as winning. On the per-benchmark tuned side, the filter must shrink the contamination gap to within statistical noise of zero on all three benchmarks. It must also keep clean-item accuracy within one point of the no-filter run. Neither the blocklist nor fixed matching may achieve both on every benchmark. The transfer and pooled ablations must lose at least one of the two properties on at least one benchmark.

Why not just…?

Why not just block the known mirror domains? This is the most serious threat. If the blocklist removes nearly all of the gap without hurting clean accuracy, content filtering is unnecessary and we will say so. There are two reasons to expect it falls short. First, copies of benchmarks migrate to blogs, forums and translated reposts, and Han et al. note that the canary strings meant to mark benchmark data are stripped when data is re-uploaded, which suggests mirrors are not confined to a fixed set of hosts. Second, blocking GitHub removes legitimate evidence for technical questions. The blocklist condition measures both effects directly. The synthetic reformatted and translated leak pages test the first effect under a controlled distribution of mirror types.

Why not just log everything and drop contaminated items afterwards? Han et al. effectively do this, and for a trusted evaluator it is simpler. It has two weaknesses. Dropping items changes the benchmark that is actually scored, and the change is biased, because the easiest-to-find items leak first. And when the developer runs the evaluation, post-hoc exclusion still depends on logs the developer controls. In-line filtering keeps every item scorable because the agent can continue with other sources, and the enclave makes the log trustworthy. We will report the filtered score on all items alongside the post-hoc score on the retained items, and check whether the two agree.

Why not just evaluate offline, or use procedurally generated tasks? Removing search changes what is being measured. DyVal generates fresh tasks and exposes memorization well, but only in formal domains such as arithmetic, logic and graph reachability. It cannot produce SimpleQA-style factual lookups, and Han et al. point out that fresh tasks do not stop agents from searching for intermediate steps.

Why not just use an ordinary Docker proxy with no enclave? If the organizer runs every evaluation, an ordinary container proxy is enough, and we will release one. The enclave is needed only when a developer runs the evaluation on their own machines with a proprietary model. In that setting, a software proxy can be silently modified. Proof-of-Guardrail reports that its enclave detected all simulated tampering attempts, a guarantee a plain container does not offer. A private scoring API run by the organizer avoids the problem only by requiring developers to hand over their model, which P⁴Bench reports developers resist.

Proposed Experiments and Decisive Checks

Leak detection offline. On held-out pools for HLE, SimpleQA and GPQA, we report recall of leak pages at a false-block rate of at most 1%, broken down by leak type. The decisive check is the transfer table. If the configuration tuned for HLE, applied to SimpleQA, either blocks answer-bearing Wikipedia pages or misses question-and-answer mirrors, task dependence is confirmed. If the pooled single configuration matches the per-benchmark ones, the representation-selection claim fails, and we will report a single universal filter instead.

Agents end to end. We run OpenHands agents on two or three LLMs under the no-filter, blocklist, fixed-matching and tuned-filter conditions. The decisive check has two parts: the contamination gap must shrink toward zero, and clean-item accuracy must stay unchanged. GPQA, which Han et al. found relatively resilient, serves as a control where the filter should have little to remove. Seeing a large drop there would indicate over-filtering.

Enclave cost. We measure proxy latency per retrieval and total wall-clock time with the enclave and without it. Proof-of-Guardrail's 25–38% overhead for a guardrail classifier is the comparison point. We expect a string-matching proxy to cost less, but we will report the measurement whatever it turns out to be.

Replay reproducibility. We rerun a sample of evaluations from the encrypted response copies and check that the resulting scores match the original live runs.