A benchmark can be "watermarked" by having a language model rephrase every question while nudging its word choices toward a secret, key-dependent half of the vocabulary. A model later trained on that text absorbs the nudge, and an auditor holding the key can detect the bias with a statistical test (Detecting Benchmark Contamination Through Watermarking). The known weakness is "data washing." If the contaminator paraphrases, reformats, or re-tokenizes the benchmark before training, the bias is mostly not erased. It is misaligned. The secret vocabulary split at each position is computed from the preceding tokens, so once those tokens shift, the auditor scores the right words against the wrong key. Hardware security has worked on this exact problem for years under the name signal desynchronization: random delays and frequency scaling smear a secret-dependent signal across time, and attackers recover it by scanning over many candidate alignments (Chameleon, ShadowScope).
We propose to treat washed contamination as a desynchronized signal. The auditor tests each suspect model under a whole family of candidate alignments: different context lengths for the hash, character- and word-level keying that survives tokenizer changes, and order-insensitive keying that survives sentence reshuffling. Scanning many alignments creates hundreds of strongly dependent tests per model, and a leaderboard audit multiplies that across models and benchmarks. We therefore combine the evidence with e-values, which are test statistics whose average under the null is at most one. Averages of e-values stay valid under any dependence, so the scan costs no correction penalty, and the leaderboard-level false discovery rate (the expected fraction of "contaminated" verdicts that are wrong) is controlled even though every test shares models, corpora, and keys. We validate the audit by actually training models on scrubbed benchmarks, not by perturbing text at test time. This is the same lesson Chameleon drew when it replaced simulated jitter with real hardware traces.
Core Claim
A contamination audit that searches over alignments and pools the results as e-values will detect benchmark contamination after paraphrase, reformatting, and tokenizer-change scrubbing at doses where the single-alignment test of Detecting Benchmark Contamination Through Watermarking loses power. It will also keep the false discovery rate at or below a nominal level across a full model-by-benchmark leaderboard without independence assumptions. The novelty is not e-values themselves, nor watermarking benchmarks. It is (i) recognizing scrubbing as a desynchronization that leaves recoverable, misaligned evidence, (ii) building alignment families inspired by side-channel resynchronization, and (iii) proving and measuring error control when the scanned tests are maximally correlated.
Prior Work and How This Differs
The work that motivates this proposal establishes both the opportunity and the gap. Detecting Benchmark Contamination Through Watermarking pre-trains 1B-parameter models on 10B DCLM tokens with injected watermarked ARC and MMLU questions. It reports one p-value per model and names paraphrase scrubbing as an open vulnerability. We keep its injection protocol and reading-mode scoring, but add scrubbing conditions, alignment scanning, and simultaneous error control.
The Emperor's New Clothes in Benchmarking? shows that surface edits keep fidelity above 0.9 while giving almost no resistance to memorization. For our purposes, this is evidence that washed text still carries most of its surface content, and so most of the misaligned watermark tokens. That paper measures contamination through accuracy shifts and offers no inferential guarantee. We supply that guarantee.
LORD-GoF controls online false discovery rate for watermark detection in document streams, holding it at 0.05 where naive thresholds reach 85%. It assumes conditionally independent null p-values and audits generated text, not absorbed training data. Our setting violates its assumption by construction.
Two papers act as constraints. Beware of counter-intuitive levels of false discoveries in datasets with strong intra-correlations shows that the Benjamini-Hochberg procedure can emit thousands of false discoveries on purely null but strongly correlated data. That is a warning against applying it naively to an alignment scan. The Benjamini-Yekutieli alternative pays a roughly $\log m$ penalty that would erase the benefit of scanning many alignments.
The method comes from the e-value literature. Hypothesis testing with e-values supplies the averaging and test-martingale tools. False Discovery Rate Control with E-values supplies the e-BH procedure, which controls false discovery rate under arbitrary dependence. A unified framework for bandit multiple testing supplies anytime-valid adaptive querying, which lets an auditor with a limited API budget choose which model to probe next.
The resynchronization ideas come from hardware. Chameleon localizes operations in obfuscated power traces with a sliding window, reaching 84 to 91% IoU. ShadowScope inserts deliberate marker spikes so that jittered GPU traces can be cut into alignable segments. We borrow both moves: scanning over offsets, and planting redundant anchors.
Research Program
The first component is the alignment family. For a watermarked benchmark, the auditor stores the key and recomputes the green/red split under $A$ candidate alignments. These include hash contexts of length one to four tokens computed in the suspect model's own tokenizer. They also include keys computed on normalized words or character n-grams, which survive tokenizer changes. Finally, they include a context-free unigram key, which survives reordering and paraphrase that preserves vocabulary. Mirroring ShadowScope's planted markers, we also change how the benchmark is watermarked: the rephrasing model embeds the bias under several keyings at once, at a measured cost in fidelity. Fidelity is evaluated with the question-level Hamming-distance protocol of the Emperor's New Clothes paper.
The second component is the test. For each alignment $a$, the auditor scores $n_a$ deduplicated (context, token) positions in reading mode. Each position gets an indicator $G_t$ that the suspect model's prediction is green. Under the null, the key is independent of the model, so $P(G_t = 1) = \gamma$, the green fraction. The per-alignment e-value is a likelihood ratio against a pre-registered alternative rate $q > \gamma$. The audit e-value is then
$$E = \sum_{a=1}^{A} w_a \prod_{t=1}^{n_a} \left(\frac{q}{\gamma}\right)^{G_t} \left(\frac{1-q}{1-\gamma}\right)^{1-G_t},$$
where the weights $w_a \ge 0$ sum to one and are fixed before seeing the data. Each product has null expectation one. Linearity then gives $\mathbb{E}[E] \le 1$, however strongly the alignments overlap. Across $m$ model-benchmark pairs, e-BH rejects the $k$ pairs with the largest e-values, for the largest $k$ such that the $k$-th largest e-value is at least $m/(\alpha k)$. This controls false discovery rate at $\alpha$ under arbitrary dependence.
The third component concerns where the null assumption can break. Deduplication handles repeated contexts. Several things can still bias $P(G_t=1)$ under the null: tokenizer artifacts, hash collisions, and models trained on other text watermarked with the same scheme. We will measure these failure modes empirically rather than assume them away.
Baselines
The naive reference point is the single-alignment test of Detecting Benchmark Contamination Through Watermarking, applied separately to each model-benchmark pair at an uncorrected threshold. On clean, unscrubbed contamination it is also the strongest existing method. It reports p = 1e-3 for a +5% contamination of ARC-Easy and p < 1e-12 after 16 exposures. We found no reported result on Lacuna for this test under paraphrase or tokenizer scrubbing, and measuring that is part of the contribution. We run it with the same suspect models, the same scored questions, and the same query budget. Beating it means matching its power without scrubbing, within noise, and exceeding it at every scrubbing condition at fixed false discovery rate.
The strongest multiplicity competitor scans the same alignment family but converts each alignment's z-score into a p-value. It then applies Bonferroni, Benjamini-Hochberg, or Benjamini-Yekutieli over all alignment and model hypotheses. Every hypothesis gets identical data and its own p-value, so the only difference from our method is how evidence is combined. To beat it, our method must keep empirical false discovery rate at or below 0.05 where Benjamini-Hochberg exceeds it. It must also show higher power than Bonferroni and Benjamini-Yekutieli.
A permutation baseline uses a max-over-alignments statistic calibrated by re-scoring each model under several hundred random keys. This is exactly valid for a single audit and is the most dangerous competitor.
Two ablations isolate the key ingredients. The first keeps e-values but uses only the original alignment. This separates the gain from scanning from the gain from e-value combination. The second keeps the full scan but drops the multi-keyed watermark. This measures whether planted anchors are needed, or whether post-hoc resynchronization suffices.
Why not just…?
Why not just use Benjamini-Hochberg across alignments and models? Neighboring alignments are near-duplicates of one another, and the same suspect model appears in every row of the leaderboard. This is the strong intra-correlation regime in which Beware of counter-intuitive levels of false discoveries documents false discoveries on null data. The null-model experiment below measures whether this happens here. If Benjamini-Hochberg turns out empirically safe in our setting, we will report that honestly, since it would make the e-value machinery unnecessary for false discovery rate control.
Why not just take the max over alignments and calibrate with random keys? This is a real threat. Key randomization gives an exact single-audit p-value and may be more powerful than a fixed-weight e-value mixture. Its weakness is composition. It does not directly support stopping as soon as the evidence is strong, or adaptive querying across a leaderboard. Turning its p-values into leaderboard-level false discovery rate control under dependence brings back the Benjamini-Hochberg-versus-Benjamini-Yekutieli dilemma. We run it head to head. If it dominates in power, we will use permutation-calibrated statistics inside each audit, converted to e-values through a calibrator, and keep e-BH only for aggregation.
Why not just simulate scrubbing by perturbing the text at test time? Training on scrubbed text and testing on perturbed text are different processes. A model learns token co-occurrences from the washed corpus, and its learned bias need not match what a test-time perturbation predicts. Chameleon found the analogous gap in hardware: simulated jitter mispredicted which defenses resist attacks. We will report both protocols. Any disagreement between them is itself a finding about how contamination detectors should be evaluated.
Why not just use post-cutoff benchmarks like LiveBench? Fresh questions prevent contamination going forward. They say nothing about whether existing leaderboard scores on static benchmarks are inflated, and they require ongoing human curation. Detection and rotation complement each other. Our audit answers the retroactive question that rotation cannot.
Proposed Experiments
The ground-truth suite follows the watermark paper. We pre-train 1B models on 10B DCLM tokens, injecting watermarked ARC-Easy, ARC-Challenge, and MMLU questions at 1, 4, and 16 exposures. Scrubbing conditions include none, LLM paraphrase, the surface-edit families from the Emperor's New Clothes taxonomy, a tokenizer change via re-segmentation, and translation round-trips. Full pre-training is costly, so we pair a small grid of from-scratch runs with a larger grid of continued fine-tuning on open 3B to 34B models. This mirrors the Emperor's New Clothes protocol, which also lets us use GSM8K, TruthfulQA, and RepliQA as additional benchmarks.
The decisive null check uses watermarked rephrasings generated with fresh keys and never released. Every production and research model is guaranteed clean with respect to these texts, yet each still shares the real dependence structure of the leaderboard. Running the full audit on these nulls gives an empirical false discovery rate for every method.
The metrics are power at a target false discovery rate of 0.05, as a function of contamination dose and scrubbing type, and empirical false discovery rate on the fresh-key nulls. We also report the fidelity cost of multi-keyed watermarking, the number of API queries needed to reach a verdict under adaptive querying, and the gap between scrubbing applied at training time and at test time.
The proposal is falsified if, under paraphrase scrubbing at 16 exposures, the alignment scan recovers no more power than the single-alignment test. That outcome would mean washing destroys the watermark rather than misaligning it. It would also show that the hardware analogy fails for text.
References
- Detecting Benchmark Contamination Through Watermarking
- Chameleon: A Dataset for Segmenting and Attacking Obfuscated Power Traces in Side-Channel Analysis
- ShadowScope: GPU Monitoring and Validation via Composable Side Channel Signals
- The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
- LORD-GoF: A Robust Online Detection Approach for LLM Watermarks in Sparse and Mixed Streams
- Beware of counter-intuitive levels of false discoveries in datasets with strong intra-correlations
- Hypothesis Testing with E-values
- False Discovery Rate Control with E-values
- A unified framework for bandit multiple testing
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark