Benchmark contamination means test questions or their answers end up in a model's training data or within reach of its web search. It is usually hunted after the fact, by probing a finished model for signs that it has seen the test. That hunt is failing. For reasoning models trained with reinforcement learning, the standard memorization detectors fall to near chance. Web-browsing agents meanwhile retrieve answer keys that were mirrored onto public sites. This proposal moves contamination control to the one moment when the evidence is cheap and unambiguous: when a human author writes the item. Annotation platforms already watch annotators live, recording agreement, timing and keystrokes. We propose to extend that live monitoring to where an item's text came from. The platform would log paste events and how far the final item drifts from pasted material, check each draft against a local copy of a large web training corpus and against a search agent, and ask the author to transform any item that is already discoverable before the effort is spent.
The underlying argument is simple. Once a model has been post-trained, nobody can reliably tell whether it saw a question. The author's editing session, however, records whether the question was lifted from somewhere. If that record predicts which items models answer without having earned it, better than any batch lookup run after export, then provenance captured at authoring time becomes a necessary part of building a benchmark, not an optional extra.
Core Claim
We claim two things. First, authoring-time provenance signals predict which benchmark items are "pre-exposed." These signals are clipboard pastes, the edit distance between pasted text and the submitted item, typing effort relative to item length, and time spent. By pre-exposed we mean items whose content or answer already exists somewhere a model could have trained on or could retrieve. The signals should add predictive power beyond what a post-hoc n-gram and search lookup of the finished dataset provides. The gain should be largest for paraphrased copies and for sources outside any open corpus we can index. Second, closing the loop improves the benchmark itself. Asking flagged authors to change numbers, recombine premises or re-derive the answer should produce items whose accuracy is less inflated by search access and by prior exposure. It should cost fewer author-hours per surviving item than discarding flagged items after export.
The first claim can fail cleanly. If a batch search script run on the exported dataset catches everything the telemetry catches, the right conclusion is "run the script." In that case the contribution shrinks to a cost argument about when items get fixed. We treat that outcome as a live possibility and design the experiments to settle it.
Why the Two Directions Need Each Other
Three findings motivate moving control to authoring time. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models with supervised fine-tuning and then applies clean reinforcement learning. It shows that the RL step shrinks the loss gap between seen and unseen questions. When advanced reasoning models are contaminated with worked solutions, all ten tested detectors average 50–58% AUROC, even though pass@1 rose by 11.7% for DeepSeek-R1-Distill-Llama-8B. Search-Time Data Contamination shows that search agents find answer keys on HuggingFace and GitHub. Roughly 3% of questions are affected, and on those items Sonar Deep Research scores 37.5% against 17.4% on clean items in Humanity's Last Exam. The same paper reports that canary strings are stripped by mirrors. Together these say that model-side auditing is unreliable and that release-time safeguards leak. What remains is controlling the item's origin.
Annotation-tool research supplies both the method and the precedent. WebAnno and SenTag compute agreement live so that administrators see guideline problems immediately. UCCAApp and EZCAT document how open-loop, export-then-check workflows discover problems only after the budget is gone. Cascading Biases is the closest precedent for our signals. It logs "heuristic traces" such as low time, low keystroke effort, and copying and word overlap with the source passage, and shows they predict artifacts that downstream models exploit. That paper studies bias in small classifiers and analyzes its traces offline. We repurpose the copying trace as a contamination signal for foundation models and act on it inside the session. No platform in the file checks drafts against training corpora or search. High agreement among annotators says nothing about whether they all copied the same web page.
Research Program
We will build on POTATO. It is a configuration-driven annotation tool that already supports attention checks and a model in the loop, and it reports that its setup cost is recouped within about 20 long documents. We add three components to it.
The first is a telemetry layer. It records paste events with their text, typing effort, dwell time, and the edit distance between each pasted span and the final submitted item. It records these on the platform only, never browser history.
The second is a local exposure index. This is MinHash and embedding search over FineWeb (resource), a 15-trillion-token Common Crawl corpus, plus known public benchmark dumps. Every draft is checked against it within seconds.
The third is a retrieval probe. A search-enabled agent receives the question with the answer withheld, and the item is flagged if the agent's retrieved pages contain the answer. There is a real risk here, which the search-time paper itself raises: sending questions to a commercial search API could leak them into query logs. We therefore run most probes against the local index and route only a small, logged subset to external search. We then later check whether the probed items became more retrievable than unprobed controls.
The authoring task is to write graduate-level science and competition-style math questions with objectively checkable answers, in the style of GPQA and AIME. We use code-verified answers following LiveBench, which avoids LLM judges because LLMs misjudge problems they cannot solve. Authors work naturally, with web access. Before any other use of the data, blinded expert auditors label each item's ground-truth exposure status by manually searching for the item's source, including textbooks and question banks outside FineWeb. A smaller seeded arm gives authors a curated library of offline, non-indexed sources. It measures how often provenance signals catch copying that no corpus lookup can see.
We also examine Gap 2 from the literature. We test whether low-effort authors, identified by the satisficing traces of Cascading Biases, are also the ones who copy exposed material. If they are, some "contamination" that detectors attribute to leakage is really an authoring-behavior effect.
Baselines
The naive reference point is the same authoring study run on unmodified POTATO. It keeps POTATO's attention checks and qualification screening and adds no provenance or exposure checks. This measures how many exposed items an ordinary careful pipeline ships.
The strongest simple competitor is post-hoc exposure filtering on the exported dataset. It uses the identical FineWeb index, benchmark dumps and search probes, run in batch after authoring ends. It is the strongest competitor because it uses exactly the same lookups as our system, minus timing and telemetry. We run it on the same items with the same thresholds, tuned on the same held-out labeled set. No reported result for this batch-lookup baseline on authored reasoning items was found on Lacuna. We therefore report its AUROC against auditor labels as our own measured reference. Beating it means a statistically significant AUROC gain from adding telemetry, with a pre-registered target of at least 0.05, concentrated on paraphrased and offline-source items.
The strongest existing model-side methods form a third group, run against open models on our items. These are Min-k% Prob and loss-based tests, PaCoST, which runs a paired t-test on confidence between original and rephrased questions, Data Contamination Quiz, which asks the model to pick the verbatim original among synonym perturbations, and ConStat, which compares accuracy on original versus rephrased benchmarks against a panel of reference models. ConStat reports detecting 98.4% of sample-specific contamination in controlled settings. The Fragility paper reports memorization detectors at 50–58% AUROC on post-trained reasoning models. We give every detector the same models and items. The comparison is item-level AUROC for pre-exposure. Our expectation is that detectors do reasonably on base models and poorly on RL-tuned reasoning models, while provenance signals are unaffected by how the model was trained.
The fourth baseline is fully automated generation with YourBench. It synthesizes questions from documents published after March 2025 (TEMPORA-0325) and reports about 85% question validity after filtering. We match its budget in dollars and compare surviving-item validity, search-time inflation and difficulty.
Ablations isolate the key ingredients. Telemetry alone runs without corpus lookup. Lookup inside the loop runs without telemetry. Flag-only lets authors see nothing, so we learn the predictive value without any behavior change. Flag-and-revise is the full system.
Why not just…?
Why not just run the same search and n-gram lookup on the finished dataset? This is the most serious threat. Batch lookup sees exactly what the local index sees. Telemetry can only add value for copying the index cannot see, such as offline sources, paraphrases below the similarity threshold, and material in proprietary training corpora. The seeded offline-source arm and the incremental-AUROC analysis settle this directly. If the gain is negligible, we will say so. The remaining case for acting in the loop is then cost, measured as author-hours per surviving valid item under revise-during-session versus discard-after-export, the same time-to-discovery argument made for live agreement in WebAnno.
Why not just audit the trained models with contamination detectors? On The Fragility of Benchmark Contamination Detection in Reasoning Models shows these detectors near chance after RL post-training. PaCoST needs token log-probabilities that commercial APIs do not expose. ConStat depends on reference models that are themselves uncontaminated. Our detector baseline measures this on our own items, split by base versus reasoning models.
Why not just generate items automatically from fresh documents or refresh the benchmark monthly? Freshness helps, but YourBench still leaves about 15% invalid items. LiveBench draws on sources that search engines progressively index, so a fresh item still becomes searchable. Human-authored hard reasoning items remain valuable, and our YourBench arm tests whether they are worth the extra cost.
Why not just use canary strings or keep the set private? The search-time paper reports that canaries are stripped by mirrors. P⁴Bench notes that privacy sacrifices public verifiability unless one adopts zero-knowledge proof machinery. Neither measure stops an author from copying an already-public problem into a private set.
Decisive Experiments
The first test is detection. On roughly 1,500 authored items with auditor labels, we compare item-level AUROC for pre-exposure across telemetry, batch lookup, their combination, and the model-side detectors. We report the results separately for verbatim, paraphrased and offline-source copies.
The second test is consequence. Items flagged by the full system should show larger search-time gains, meaning accuracy with search enabled minus accuracy without it, measured on the Sonar agents used in Search-Time Data Contamination and on an open search agent. Flagged items should also show higher model accuracy relative to difficulty estimated from blinded human solvers. We compare this model-side inflation on unflagged items against items caught only by batch lookup.
The third test is intervention. We randomize authors to flag-only or flag-and-revise. Revised items should show search-time gain and exposure-driven inflation indistinguishable from never-flagged items, while keeping validity. Expert-judged validity should be at or above YourBench's reported 85%. Author-hours per surviving item should also be lower than under discard-after-export.
The fourth test is robustness. We follow the Fragility protocol: fine-tune an open model with supervised learning on a subset of our items and then apply GRPO. The GRPO step should drop the model-side detectors toward chance, while the provenance record, fixed when the item was written, still identifies which items could have been seen.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Search-Time Data Contamination
- WebAnno: A Flexible, Web-based and Visually Supported System for Distributed Annotations
- SenTag: A Web-Based Tool for Semantic Annotation of Textual Documents
- UCCAApp: Web-application for Syntactic and Semantic Phrase-based Annotation
- EZCAT: an Easy Conversation Annotation Tool
- Cascading Biases: Investigating the Effect of Heuristic Annotation Strategies on Data and Models
- POTATO: The Portable Text Annotation Tool
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Yourbench: Dynamic Evaluation Set Generation with LLMs
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs