Does Data Curation Concentrate Benchmark Leakage?

Training-data curation methods now routinely beat full-dataset training with a small fraction of the data. A curated 6% of an instruction set outperforms the whole set, and semantically pruned web text trains models faster and scores higher downstream. These methods pick data that looks "high-value" to an automatically learned scorer. Leaked benchmark questions are an obvious candidate for looking high-value. They are clean, well-formed, dense with answer-bearing content, and in some methods they are scored directly against the very benchmark used for evaluation. Separately, we know that a model which has merely seen a leaked test item gains almost nothing from it, while one that has seen it many times exploits it heavily. If curation retains leaked items and then trains on them for repeated epochs, part of the celebrated "quality over quantity" gain could be measurement error.

We propose a controlled audit to test this. We plant benchmark items into curation pools at known rates and in graded disguises, built with lexical resources: verbatim copies, WordNet synonym rewrites, dictionary-guided French renderings, and full translations. We then run real curation selectors over the pools and measure two things: how much each selector enriches the planted items, and how much accuracy the planted items buy on the benchmark relative to matched unplanted twins. The result is a per-selector "leakage amplification" number, plus a test of whether standard decontamination survives each disguise.

Core Claim

Curation selectors differ systematically in whether they amplify or suppress contamination. The direction depends on what the selector's learned features reward. We expect a Shapley-style selector whose utility is computed on a target benchmark to enrich planted items of that benchmark well above their base rate. We expect a difficulty-aware filter that keeps items a reference model fails to deplete verbatim leaks, since a reference model that has also seen them finds them easy. We expect embedding-based semantic deduplication to collapse verbatim and synonym copies, which lowers the duplication that drives exploitation, while failing to collapse cross-lingual copies that its embedder does not place near one another. Both outcomes are possible and neither has been measured. The new contribution is the measurement: enrichment and exploitation per selector, per disguise level, with ground truth known by construction.

The four directions play different roles. Data curation is the object under study. Benchmark contamination is the failure being measured. Automated feature extraction appears as the mechanism, because every selector here acts through a learned embedding or proxy-model score, and the embedder's invariance to surface change decides what survives. Lexical resources supply the method: they let us build disguised leaks whose meaning is preserved and whose surface distance is controlled.

Prior Work and Positioning

The motivating evidence comes from three papers that are rarely read together. Data Contamination: From Memorization to Exploitation pre-trains BERT models with planted test items. It finds under 1% exploitation for single exposures but up to 40% when an item is duplicated 200 times, so redundancy is the lever. That study used masked language models and uncurated mixtures. We add a curation step between planting and training, and we use generative models.

D4 prunes web text with OPT-125M embeddings, semantic deduplication, and prototype selection, then trains for multiple epochs on the result. Its authors note that web validation perplexity got worse because the validation data shared the templated junk that was pruned. That observation shows curation interacts with evaluation data in ways that are easy to misread. D4 never audits whether benchmark items were retained.

SHED scores data clusters by proxy Shapley values computed against downstream benchmarks and reports large gains from tiny subsets. Because its scorer looks at the benchmark, it is the selector most at risk of pulling leaks inward. DART-Math allocates sampling by reference-model fail rate. It gives us a selector whose bias plausibly points the other way.

Two papers constrain the design. Data Contamination Can Cross Language Barriers shows that translated benchmarks inflate English scores while evading n-gram and perplexity detectors. Any decontamination baseline must therefore be tested against translated disguises, not only verbatim ones. Its Choice Confusion probe gives us a detector to evaluate. Benchmarking Benchmark Leakage in Large Language Models shows paraphrase-based perplexity detection breaks under rephrasing. We treat that as a reason to rely on planted ground truth rather than post-hoc detection.

The lexical method draws on Inducing Embeddings for Rare and Unseen Words by Leveraging Lexical Resources, which uses WordNet graph structure to move between words without corpus evidence, and on the Lefff-based tagging work of Denis and Sagot. We use those resources to generate morphologically correct French substitutions. In both papers lexicons serve to improve a model. Here they serve to build controlled disguises.

Research Program

We split each target benchmark into two matched halves, balanced by subject and baseline difficulty. Planted items come from one half; the other half is the clean twin. The benchmarks are MMLU, ARC-Challenge, GSM8K, and MathQA. Each planted item appears at one of four disguise levels, ordered by increasing surface distance at roughly fixed meaning:

  1. A verbatim copy, either question-only (the survey's Data Level) or question plus answer (the Label Level) in Benchmark Data Contamination of Large Language Models: A Survey.
  2. A WordNet synonym rewrite, with content words replaced by synset neighbours and answer tokens kept.
  3. A dictionary-guided French rendering, with lexical substitution and Lefff inflection but English syntax.
  4. A full machine translation into French, as in Yao et al.

Each disguised copy is planted at duplication counts of 1, 10, and 100.

We use two curation regimes. In the pre-training regime, a pool of a few billion FineWeb tokens (HuggingFaceFW/fineweb) receives planted documents. D4, D4 without prototype selection, and random subsampling then each select the same token budget, and we train 125M–1B decoder models for multiple epochs on each selection. In the instruction-tuning regime, WizardLM-evol-instruct-70k receives planted question-answer pairs, and SHED (scored against the target benchmark, as in the original), a fail-rate difficulty filter in the style of DART-Math, and random selection each pick 4,000 and 9,000 examples to fine-tune a 7B base model.

We measure three quantities. The enrichment ratio is the fraction of planted items in the selected subset divided by their fraction in the pool. Exploitation is accuracy on the planted half minus accuracy on the clean half, against the same gap for a model trained on the unplanted pool. Embedder invariance is the cosine similarity, in each selector's own embedding space, between an original item and its disguised copy. Embedder invariance is the mechanistic link: it should predict which disguise levels deduplication collapses.

Baselines

The naive reference point is random selection at the same budget from the same planted pool, trained with identical steps, epochs, and hyperparameters. By construction it keeps planted items at their base rate. Every enrichment and exploitation figure is reported relative to it.

The strongest existing curation results are the methods themselves, rerun under our controls. For SHED, the reported figure is 35.92% for a 4,000-sample subset versus 33.16% for the full 70k WizardLM set (SHED), and 49.21% versus 45.99% on ARC-Challenge with 9,000 selected samples. For D4, the reported figure is an average 0-shot gain of up to 2.04% across 16 benchmarks at 6.7B (D4). Our scale is smaller, so we do not expect to reproduce those absolute numbers. We first reproduce each method's gain over random selection on an unplanted pool, at our scale and with equal tuning budgets, so that any later effect is not an artifact of a broken reimplementation.

The strongest decontamination baseline is n-gram overlap filtering against the benchmark, applied to the pool before curation. We also test a stronger variant that filters by embedding similarity to benchmark items. For detection after training, we compare against Choice Confusion. Yao et al. report a +26.25-point gain for clean LLaMA3 versus roughly a 17-point drop for contaminated models on MMLU. We ask whether it still separates our models when contamination arrives through curation rather than direct fine-tuning.

The key ablation removes benchmark access from SHED's utility scorer by scoring on a disjoint dev set from a different benchmark. This isolates whether enrichment comes from benchmark-targeted scoring or from generic "quality" features.

A selector counts as amplifying contamination if its enrichment ratio exceeds 1 with a bootstrap confidence interval excluding 1, and its exploitation gap exceeds random selection's at matched base rate. A decontamination method counts as sufficient if, at every disguise level, it brings exploitation within noise of the unplanted pool.

Why not just…?

Why not just run n-gram decontamination before curating? For verbatim and synonym leaks this may suffice, and we expect to confirm that. Yao et al. show translated benchmarks pass n-gram and perplexity checks while raising English MathQA from 42.01% to over 95% for LLaMA3-8B. Disguise levels 3 and 4 test exactly that case. If n-gram filtering removes all measurable exploitation at every level, that answer is still worth publishing, but we do not expect it at level 4.

Why not just evaluate curated models on LiveBench or DyVal, which resist contamination? This is a real threat. If D4 or SHED gains carry over unchanged to LiveBench, contamination cannot be what explains them. We run this check. It is a weak substitute for the audit, though. These benchmarks cover mostly math, code, and logic (DyVal), so they say nothing about MMLU-style knowledge, where SHED's largest gains are reported. They also show only that a gap exists, not which selector produced it or why.

Why not just filter by embedding similarity to the benchmark? This is the most plausible simple fix. It would reduce the proposal to an audit that ends by recommending it. We take that seriously: the embedding filter is a baseline, and the embedder-invariance measurements predict where it fails. An English-centric embedder such as OPT-125M should place full translations far from their originals. If it does not, and the filter catches all four levels, we report embedding decontamination as the recommended default. The contribution then shrinks to the quantified enrichment results, which are still missing from the literature.

Decisive Checks

Three outcomes settle the question.

The first concerns SHED. If its enrichment ratio for target-benchmark items is significantly above 1, and the ablation without benchmark access brings it back near 1, then benchmark-scored curation pulls in leaks and its reported gains need re-evaluation on held-out halves.

The second concerns D4. If its exploitation at duplication 100 falls below random selection's for levels 1–2 but not for level 4, semantic deduplication protects against surface leaks while multi-epoch training over surviving translated copies recreates the duplication effect of Magar and Schwartz. That would be a concrete, actionable failure.

The third concerns the clean twins. If, for every selector, the planted-half advantage is within noise of the unplanted control, the curation gains are real, and the field gains a reusable planted-pool protocol for checking that claim about any future selector.

Across all three, we report embedder invariance alongside enrichment, to test whether one cheap embedding measurement predicts a selector's contamination behaviour before any model is trained.