Benchmark creators increasingly ship test sets in encrypted archives so that web crawlers cannot scoop them into language-model training data. Encryption only controls who can open the file, though. Once any authorized researcher decrypts it and pushes the plain text to GitHub, the protection is gone for everyone, permanently. Nobody can tell whose copy leaked, and nobody can tell whether a particular model was trained on it. We propose to give every recipient a slightly different copy of the test set. Each copy is built by choosing, item by item, among several interchangeable wordings, and the pattern of choices spells out a code unique to that recipient. If a copy later shows up on the web, or inside a model, the code identifies it.
The difficulty is that standardized evaluation depends on everyone scoring the same test. A fingerprint that changes which questions a model gets right would trade contamination for irreproducibility. The proposal therefore rests on two linked contributions. The first is a procedure that certifies the wording variants as score-equivalent at the level of individual questions, not only in aggregate accuracy. The second is a detection test that takes advantage of those same interchangeable variants: because no variant is favored before any leak occurs, a model's preference for one recipient's wordings yields an exact statistical test for training on that copy, with no reference model and no calibration.
Core Claim
We claim that choosing among certified-equivalent item variants according to a collusion-resistant code does two things at once. It makes leaked benchmark copies traceable to their recipient, both as text found on the web and as memorization inside a trained model. It also leaves question-level scores statistically indistinguishable across copies, so a standardized harness can report a single comparable number.
The detection statistic works as follows. Recipient $k$ receives codeword $c_k \in \{1,\dots,r\}^n$, meaning variant $c_{k,i}$ of each item $i$ out of $r$ variants. For a suspect model with per-token loss $\ell(i,v)$ on variant $v$ of item $i$, the score is
$$ S_k = \sum_{i=1}^{n} \Big( \tfrac{1}{r}\textstyle\sum_{v} \ell(i,v) - \ell(i, c_{k,i}) \Big). $$
$S_k$ measures how much more familiar the model finds recipient $k$'s wordings than the item's average wording. Under the null hypothesis that the model never saw copy $k$, the codeword is independent of the model, and the variants were drawn exchangeably. The distribution of $S_k$ is then given exactly by re-drawing random codewords. The paired, within-item design cancels item difficulty, which is the main nuisance that makes absolute-loss membership tests noisy.
Relation to Prior Work
The motivating gap comes from Stop Uploading Test Data in Plain Text. That paper proposes password-protected archives with no-derivatives licenses. It concedes that one decrypting user who re-uploads spoils the benchmark, and it measures heavy off-platform "shadow usage": 4.2 citing models per official leaderboard submission (Jacovi et al.). Those uncontrolled local copies are exactly the objects we fingerprint. The security literature frames the leak as the gap between access control and usage control (Rahmani et al.; Garcia et al., where a pharmacist with delegated access simply copies the decrypted prescription). The standard answer to that gap is traitor tracing, which embeds a recipient-specific mark rather than tracking keys (Fiat et al.). We borrow that idea and its concern with colluding recipients, and we apply it to text that will be absorbed by a model rather than read by a person.
The closest method is Detecting Benchmark Contamination Through Watermarking. It rephrases every question with one secret-keyed "green-list" token bias and tests a suspect model for that bias. It answers whether a model saw the benchmark, not whose copy. Its guarantee also depends on free-form rephrasing, which risks changing constraints in math or code. We differ in three ways. We encode identity rather than presence. We use discrete variant selection with an exact permutation null instead of a token-bias z-test. And we make score invariance a certified property rather than an aggregate observation.
That last point is forced on us by The Emperor's New Clothes. It shows that stable aggregate accuracy can hide large question-level churn, measured by Hamming distance, and that surface edits keep fidelity above 0.9 while aggressive ones drop MMLU fidelity to 0.686. The same paper constrains our design, because it shows surface variants give contaminated models almost no resistance. We turn that weakness into the property we need. If a memorizing model still recognizes a surface variant as close to the original, it will also recognize its own recipient's variant more strongly than the others, and that preference is our signal.
P⁴Bench avoids answer leakage entirely by never publishing answers and verifying zero-knowledge proofs instead. It works only for tasks that can be checked by an arithmetic circuit, such as satisfiability and discrete logarithm, and it leaves questions public. We target the multiple-choice and short-answer natural-language benchmarks it cannot cover. Network-isolated "storage capsules" (Borders et al.) would prevent leaks outright, but they require every evaluator to run inside controlled virtual machines, which cuts against the local-copy practice Jacovi et al. document.
Research Program
The first component generates and certifies variants. For each item we produce $r$ candidate wordings, and we try three families that trade detectability against fidelity. Answer-option reordering carries no semantic change. Minimal surface edits cover synonyms, clause order, and number formatting where the value does not change. The third family is constrained rephrasing checked by a verifier model. A variant enters the pool only if a panel of clean models, trained without the benchmark, gives it statistically the same per-question correctness as the original. For the certification test we use exact McNemar-style tests pooled across models, together with the fidelity metric and normalized Hamming distance from The Emperor's New Clothes. Items without enough certified variants carry no code bits, which costs capacity but not validity.
The second component is the code itself. We assign codewords from a collusion-resistant fingerprinting code of the Tardos type, so that a coalition merging several copies still implicates at least one member with bounded false-accusation probability. We then study how code length $n$ and alphabet size $r$ trade off against the number of recipients.
The third component is detection in two modes. Text mode matches web-found or dataset-found copies against codewords, which is nearly trivial. Model mode computes $S_k$ from token losses under the permutation null. For API-only models without per-token losses, we evaluate a weaker variant that uses the model's choice among variants presented side by side, and we report the power lost.
The fourth component is harness integration. A standardized loader in the style of Datasets issues the fingerprinted copy on authenticated download, keeps the recipient-to-codeword registry, and maps every variant back to its canonical item ID so that reported scores refer to one shared benchmark.
Baselines
The naive reference point is the Jacovi-style encrypted archive with one identical plaintext for all recipients. It gives zero traceability by construction and serves as the zero-fidelity-cost anchor for the invariance comparisons.
The strongest existing competitor is the green-list watermark of Detecting Benchmark Contamination Through Watermarking, extended to give each recipient its own secret key. This is the obvious way to repurpose it for tracing. It is the real threat because its detection on a single copy is already very strong. The paper reports a p-value of $10^{-3}$ at a +5% score inflation on ARC-Easy, and below $10^{-12}$ after 16 exposures of ARC-Easy, for 1B models trained on 10B tokens of DCLM. We run it on the identical pretraining corpora, contamination schedules, and suspect models, with the same number of recipients. Each method gets the same budget for tuning variant strength (bias δ for them, variant family and $r$ for us), selected on a held-out benchmark. We beat it if we match its single-leaker detection power at matched fidelity while achieving lower false accusation of innocent recipients and correct identification under collusion by two to four recipients, where per-key green lists blend together. We also count it as a win if we reach equal power with measurably less question-level churn.
A per-recipient canary string, meaning a unique identifier embedded in each copy, is the cheap tracing baseline. The ablations isolate our two key ingredients. The first swaps certified variants for uncertified random variants, which tests whether certification matters for fidelity. The second swaps the paired within-item statistic $S_k$ for an absolute-loss membership score on the recipient's copy, which tests whether the exchangeable design is what gives detection power.
Why not just…?
Why not just give each recipient their own watermark key? This is the most plausible alternative, and we treat it as the main baseline rather than dismissing it. Our expectation that it falls short rests on two points. Colluding recipients who average or splice copies produce text carrying partial biases from several keys, and testing many keys inflates false alarms. Its rephrasing also changes more text per item, and the question-level analysis in The Emperor's New Clothes suggests that is where hidden churn comes from. If the per-key watermark matches us on both collusion and churn in our experiments, the honest conclusion is that its test plus a registry suffices, and our contribution shrinks to the certification procedure.
Why not just embed a unique canary string per copy? The watermarking paper found that models often fail to memorize isolated canary strings, and cleaning pipelines strip unusual random strings. Canaries also sit apart from the questions, so they are trivial to delete before re-upload. Our fingerprint lives in the wording of the items themselves and cannot be removed without re-editing every question. We include canaries as a baseline to measure this directly.
Why not just keep the test set private behind an API? P⁴Bench and Jacovi et al. both note that private leaderboards demand trust in a central organizer, are resource-heavy, and send prompts through commercial APIs that may train on them. The shadow-usage ratio shows that the community keeps local copies anyway. Our approach assumes that behavior will continue and makes it accountable.
Why not just procedurally regenerate the benchmark for every run? This works for rule-governed tasks such as A Cognitive Battery for Foundation Models. It does not work for expert or commonsense question answering, and fresh instances every run break the cross-run comparability that standardized evaluation exists to protect.
Proposed Experiments
For contamination, we follow the controlled setup of the watermarking paper: 1B-parameter models trained from scratch on 10B tokens of DCLM, with benchmark copies injected at 1, 4, and 16 exposures. We fingerprint ARC-Easy, ARC-Challenge, and MMLU, and add GSM8K to stress fidelity where wording changes can alter the math. We simulate 100 recipients. In each run, one recipient's copy, or a merge of 2 to 4 copies, is leaked into pretraining.
The primary metrics are three. We measure the rate of correctly identifying the leaker at a fixed false-accusation rate of $10^{-3}$ per innocent recipient. We record the p-value as a function of score inflation, for direct comparison with the reported $10^{-3}$ at +5% on ARC-Easy. And we measure fidelity and normalized Hamming distance between copies, evaluated on clean pretrained models of several families (for example Llama, Qwen, and Yi checkpoints, as in The Emperor's New Clothes) that were not used in certification. The fidelity target is to reach or exceed the >0.9 level of surface edits, with no significant question-level churn after correcting for multiple comparisons.
Robustness tests apply the post-leak "washing" attacks that the watermarking paper leaves open: model paraphrasing of the leaked copy, round-trip translation, and option reshuffling. For each attack we report how much of the detection signal survives. We expect option reordering to be erased by reshuffling, while wording variants should survive partly. We will report where the method breaks rather than claim robustness to paraphrase.
Three outcomes would decide the matter. The central claim fails if certified variants cannot reach detection power comparable to the green-list baseline at 4 or more exposures. It also fails if the permutation null proves miscalibrated on genuinely clean models, which we check by computing $S_k$ for codewords never issued and verifying a uniform p-value distribution. Finally, it fails if the per-key watermark identifies colluders as well as the Tardos code does. Any one of these outcomes would tell us which half of the design, the certification or the code, is carrying the result.
References
- Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
- A New Robust Watermarking Scheme to Increase Image Security
- Blockchain-Aided and Privacy-Preserving Data Governance in Multi-Stakeholder Applications
- Tracing Traitors
- Detecting Benchmark Contamination Through Watermarking
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs
- Protecting Confidential Data on Personal Computers with Storage Capsules
- Datasets: A Community Library for Natural Language Processing
- A Cognitive Battery for Foundation Models: Theory-Grounded Benchmarks for Attention, Learning, Metacognition, Executive Function, and Social Cognition