Do Data Curricula Launder Benchmark Leakage?

Many language-model training pipelines no longer shuffle data uniformly. Instead they use a curriculum: they sort training text by how easy a reference model finds it, and they feed it to the model in a planned order. The standard easiness signal is perplexity, meaning how surprised the reference model is by the text. The trouble is that leaked benchmark test items, the questions and answers that slipped into web scrapes, are exactly the kind of text a reference model finds unusually unsurprising, especially if that reference model saw them too. A perplexity-sorted curriculum will therefore not scatter leaked items at random. It will pile them into one stretch of training. Prior controlled studies show that when a model sees leaked items changes how much it cheats. So a curriculum's reported benchmark gain may partly come from where it happens to put the leaks, and not from better learning.

We propose a controlled pretraining study that measures this directly. We inject known test items into a pretraining corpus, run random ordering and several curricula over the same tokens, and split every benchmark into a leaked half and a never-seen half. That lets us separate the part of a curriculum's gain that carries over to unseen items from the part that is memorized leakage. To summarize real capability across tasks we use the harmonic mean of never-seen accuracy. The harmonic mean is an average dominated by the weakest tasks, so it cannot be raised by large gains on a few leaked benchmarks. The result should tell practitioners whether curricula launder contamination into apparent capability, and it tests a simple fix: take suspiciously easy items out of the curriculum and place them at random positions.

Core Claim

A difficulty-based curriculum works as an uncontrolled contamination-placement policy. The direction of its effect cannot be predicted from current evidence, so it has to be measured.

Two findings pull in opposite directions. Data Contamination: From Memorization to Exploitation injected labeled test items into BERT pretraining at different stages. It found that early exposure leads to more downstream exploitation than late exposure, and that exploitation grows with duplicate count, reaching 40% at 200 copies. This paper motivates the proposal. Classic easy-first curricula, however, put low-perplexity text first. FRAME, in contrast, stages pretraining from high to low perplexity across four quadrants. It reports a 16.8% average improvement over random ordering on MMLU and CMMLU at 3B scale and never audits for leakage. Under FRAME, leaks would land late, closest to evaluation. The Impact of Post-training on Data Contamination shows that 25B tokens of later pretraining can hide a contamination gap and that clean supervised fine-tuning brings it back. Recency and resurfacing therefore both matter too.

Our claim has two parts. First, the ordering a curriculum imposes changes how much leaked items are exploited, by an amount comparable to the curriculum's clean-data benefit. Second, this effect disappears when leaked items are re-randomized while everything else stays sorted.

Research Program

The testbed follows Investigating Data Contamination for Pre-training Language Models. That paper pretrained GPT-2 models (124M and 774M parameters) from scratch on a 3.3B-token Pile subset with controlled leakage. It found that leaking ground-truth answers matters far more than leaking the input text alone, and that performance rises and then falls as repetitions increase. We reuse this setup as our method source.

We take SST-2, SST-5, SNLI, MMLU, SQuAD, and CNN/Daily Mail and randomly split each test set in half. One half is injected into the corpus in prompt-plus-answer form at a fixed repetition count (5 and 10, below the point where Jiang et al. saw repetition start to hurt). The other half is never seen. A third arm injects paraphrased versions of the leaked half, generated with the probing transformations in Dynamic Evaluation of Large Language Models by Meta Probing Agents via promptbench. This mimics the implicit contamination that n-gram filters miss.

The first analysis is purely descriptive. Before any model is trained, we compute where each curriculum places the injected items in the training sequence. We use two scoring conditions. In one, the reference models are trained on the clean corpus. In the other, they are trained on the contaminated corpus. The second condition is the realistic case, because public reference models have probably seen these benchmarks.

For each task $i$, let $L_i$ be accuracy on the leaked half and $U_i$ accuracy on the unseen half. The exploitation gap is $e_i = L_i - U_i$. Capability across the $n$ tasks is summarized as

$$H_U = \frac{n}{\sum_{i=1}^{n} 1/U_i},$$

the harmonic mean of unseen-half accuracy. We use the harmonic mean because it tracks the weakest tasks, following How not to Lie with a Benchmark and BIG-Bench Extra Hard. All metrics are accuracy or ROUGE, which are strictly positive, so the harmonic mean's breakdown at zero never arises. A curriculum's gain over random ordering on the usual arithmetic mean of full-test scores then splits into a change in $H_U$ and a change in mean $e_i$. Laundering means the second change is large relative to the first.

The fix is contamination-exempt pacing. Items flagged as suspiciously easy are removed from the sort and spread uniformly over training. An item is flagged if its reference perplexity is low for both the weak and strong reference models, or if sampled continuations collapse onto one string, which is the output-distribution signal behind CDD in Generalization or Memorization. All other data stay in curriculum order.

Baselines

The naive reference point is uniform random shuffling over the identical contaminated corpus, with the same tokens, model sizes, optimizer, learning-rate schedule, and three seeds. We also train a clean-corpus random-order model to anchor $U_i$.

The strongest existing method is FRAME's four-quadrant schedule, which orders data by perplexity and by the perplexity difference between a small and a larger reference model, with S-shaped transitions between stages. We reimplement it with 100M and 1.3B-scale reference models scaled down proportionally for GPT-2. It gets the same token budget and the same small sweep over transition sharpness that we give our own variant. Its paper reports 16.8% average improvement over random on MMLU and CMMLU at 3B, a 15.3% absolute MMLU gain. There is no reported result on Lacuna for FRAME at GPT-2 scale or under controlled contamination. The comparison anchor from Jiang et al. is MMLU accuracy of 24.3% for the ground-truth-contaminated GPT-2 large at 10 repetitions versus 22.8% clean.

We add a plain easy-first perplexity curriculum, which puts leaks early, as the direct counterpart to FRAME, which puts them late. We also add a batch-clustering arm in the style of CommonIT. CommonIT groups instruction data into homogeneous batches, and it would concentrate leaked items into all-leak batches.

The key ablation is timing-matched random ordering. Leaked items are placed at exactly the positions a given curriculum would assign them, and all other data are shuffled. This isolates placement from the curriculum's effect on clean data.

Our fix beats the baselines if it matches FRAME's $H_U$ within seed noise while reducing mean $e_i$ to the level of random shuffling or below.

Why not just…?

Why not just run n-gram decontamination before training? In the verbatim-injection arm this would largely work, and we report it as an extra baseline. Jiang et al., however, found that the n-gram filters used in PaLM and Llama left contaminated models with high performance. CDD's authors show that paraphrased leaks evade such filters. We test this in our paraphrase arm by applying 13-gram filtering first and measuring the exploitation that remains under each curriculum.

Why not just vary injection timing under random order, as Magar and Schwartz did? This is the most serious threat. If the timing-matched ablation reproduces every curriculum's exploitation gap, then "curriculum effect equals timing effect" and our mechanism reduces to theirs. That outcome is still useful, because it gives a way to predict leakage exploitation from a curriculum's sort key alone. It also replicates the early-bird result, previously shown only in BERT with fine-tuning, in autoregressive models. If instead curricula exceed the timing-matched ablation, the extra exploitation comes from clustering or from interaction with the curriculum's shaping of clean data. The CommonIT arm tests the clustering explanation.

Why not just drop the suspiciously easy items instead of re-randomizing them? The flag also catches legitimately easy text. CommonIT reports that organizing data beats filtering it out. We run a drop arm to see what $H_U$ it costs.

Why not just use the gap between arithmetic and harmonic means as a contamination detector? Tasks vary in native difficulty, so this gap is confounded. We report it for each arm to see whether it tracks mean $e_i$, but we treat it as a secondary outcome, not the measure.

Proposed Experiments

The decisive checks are as follows. First, we measure where each curriculum places leaked items under clean versus contaminated reference scoring. We expect contaminated reference models to push leaks sharply toward the low-perplexity end.

Second, we compare mean $e_i$ across random, easy-first, FRAME, clustered, and timing-matched arms at both model sizes. If Magar and Schwartz's early-bird effect carries over to autoregressive models, easy-first should show the largest gap. FRAME's late placement could go either way, since recency may matter more without a fine-tuning stage.

Third, we check whether FRAME's arithmetic-mean advantage over random survives in $H_U$. If most of it vanishes, that is direct evidence of laundering.

Fourth, following The Impact of Post-training on Data Contamination, we apply a short clean supervised fine-tuning stage to every checkpoint and re-measure $e_i$. This tests whether late-placed leaks that look hidden come back.

Fifth, we test whether contamination-exempt pacing keeps the curriculum's $H_U$ while removing the extra exploitation.

The study is falsified if all curricula produce $e_i$ within seed noise of random ordering, if their gains show up equally in $H_U$, or if both hold together.