When a benchmark question leaks into a language model's training data, it rarely arrives as a clean copy. It shows up reformatted on a forum, with typos from a scrape, as a translated study guide, or reworded in a tutoring blog. Two recent results seem reassuring. Models largely forget verbatim leaked test items once they keep training on enough fresh text, and simple decontamination filters remove verbatim copies easily. Neither result says anything about the near-copies. This proposal measures how much test-score inflation a leaked item buys as a function of how far it sits from the original in edit distance, the minimum number of word insertions, deletions, or substitutions needed to turn one string into another, and how quickly that inflation fades as clean training data grows. The output is a "contamination kernel": a map from edit distance and clean-token volume to residual score inflation.
The kernel matters because it answers a concrete question that dataset builders currently settle by guesswork: how loose should an approximate string-matching filter be before training? If inflation drops to zero beyond a small edit radius, cheap fuzzy filters are enough. If reworded copies inflate scores as much as verbatim ones and also resist forgetting, then filters based on string distance are fundamentally limited, and "train on more data" offers no protection against the most common form of leakage.
Core Claim
We claim that the benefit a model gets from a leaked benchmark item, and how long that benefit survives continued training on clean data, can be summarized by a single smooth function of the item's normalized token edit distance to the test item and of the clean tokens seen after exposure. We predict this function decays non-monotonically. Lightly perturbed copies, with typos, formatting changes, or a few swapped words, behave like verbatim copies in the boost they give. Heavily reworded copies give a smaller boost that is harder to forget, because the model has to absorb them as content rather than as a string. If this holds, the edit radius a decontamination filter must cover is set by where the kernel reaches zero, which we can measure directly instead of inheriting a threshold such as the METEOR > 0.75 cutoff used in An Open-Source Data Contamination Report for Large Language Models.
What Is Known and What Is Missing
Three controlled pretraining studies motivate the work, and all three inject only verbatim copies. How Much Can We Forget about Data Contamination? pretrains 124M–1.6B models on FineWeb-Edu with verbatim benchmark items repeated 4× to 144×. It shows that the contamination gain fades with clean data: continued pretraining of OLMo-7B for 13% of its schedule removed 87% of the gain. The authors explicitly name paraphrased "soft contamination" as open. Quantifying the Effect of Test Set Contamination on Generative Evaluations injects up to 3,162 verbatim MATH replicas into 34M–344M models. It finds near-perfect scores that collapse to near zero on rephrased or perturbed problems. That study varies the evaluation text, not the training text, so it cannot say whether training on a paraphrase transfers to the original. Investigating Data Contamination for Pre-training Language Models and its companion Does Data Contamination Make a Difference? inject verbatim text and ground-truth contamination into GPT-2 models. They find a gain that peaks near 10 repetitions, and that n-gram decontamination filters leave inflation in place. They never test edit-distance filters. We reuse the injection-and-pretrain methodology of these papers and change the one variable they all hold fixed: the distance between leaked text and test text.
The detection side constrains the design rather than motivating it. Generalization or Memorization detects contamination in finished models by sampling many completions and measuring edit distances between them, a method called CDD. It shows that rephrased leaks also produce peaked output distributions. This tells us soft contamination leaves a trace, but CDD is a post-hoc detector and says nothing about training dose. Infini-gram mini audits 83TB of text with a compressed exact-match index, finding 74.2% of GSM8K in July 2025 crawls and 27.7% of MMLU in DCLM-baseline. It supports only exact matches, so it gives a lower bound on leakage. The approximate-matching literature, from normalized edit distance in The Influence of Minimum Edit Distance on Reference Resolution to suffix-array search in Fast Approximate String Matching with Suffix Arrays and A* Parsing, supplies the distance measures and the seed-then-verify search strategy we use for filtering at scale.
Research Program
The first component builds a graded leakage set. For each test item in ARC-Easy, ARC-Challenge, MMLU, and GSM8K, we generate variants spanning a controlled range of normalized token edit distance $d \in [0,1]$ to the original question-plus-answer text. The variant families are character noise mimicking scraping and OCR errors, formatting and template changes, word-level substitutions and deletions at fixed rates, sentence reordering, full LLM paraphrase, and, for GSM8K only, number-perturbed versions with recomputed answers as a negative control. Each variant is binned by its measured $d$, not by its generation method. Generation method is kept as a covariate, so we can check whether edit distance alone explains the effect.
The second component is controlled pretraining. Following the setup of How Much Can We Forget about Data Contamination?, we train decoder models at 124M, 350M, and 1.6B parameters on FineWeb-Edu, described in The FineWeb Datasets. Disjoint random halves of each benchmark serve as contaminated and clean controls within the same run. Each contaminated item receives variants from exactly one distance bin at a fixed repetition count of 4×, 16×, or 64×. Injection happens in a fixed window early in training, so forgetting can be read from checkpoints. Define the gain $G(d, n, T)$ as accuracy on contaminated originals minus accuracy on matched clean items, for distance bin $d$ and repetitions $n$, measured after $T$ clean tokens have been seen since injection. We fit
$$G(d,n,T) \approx A(d,n)\,\exp\!\big(-T / \tau(d)\big),$$
where $A$ is the initial boost and $\tau$ is a forgetting scale in clean tokens. The exponential form follows the cumulative weight-decay argument in the forgetting paper. Our central hypotheses concern the shapes of $A$ and $\tau$ in $d$. Because small models sit near chance on MMLU (Measuring Massive Multitask Language Understanding), we also report a likelihood-based gain, the drop in loss on the correct answer, as a more sensitive secondary measure.
The third component turns the kernel into a filter and tests it. We implement approximate decontamination that finds candidate documents through short exact n-gram seeds and then verifies them with banded token edit distance at radius $r$. The method is affordable on a FineWeb-Edu-sized corpus because the expensive alignment runs only on seed hits. We choose $r$ from the kernel: the smallest distance at which $A(d,n)$ is statistically indistinguishable from zero. We then retrain on a contaminated corpus filtered at that radius.
Baselines
The naive reference point is doing nothing: training on the contaminated corpus unfiltered, with no extra clean data. This sets the ceiling of inflation. The matched clean model, trained on the same corpus with all variants removed by construction, sets the floor that any filter should reach. Every arm shares the same base corpus, token budget, model sizes, optimizer, learning-rate schedule, and evaluation harness. We use the corrected LaTeX answer checking noted in Quantifying the Effect of Test Set Contamination on Generative Evaluations, with three seeds per arm.
The strongest existing mitigation is simply continued clean training, as quantified in How Much Can We Forget about Data Contamination?. Their reported number is that continued pretraining of OLMo-7B for 13% of its remaining schedule removed 87% of the contamination gain, for verbatim items. We run it by extending clean training after injection up to 5× the compute-optimal token count, matching their 124M finding. Beating it means showing that, for some distance range, the 87%-style decay fails to appear: residual gain stays statistically above zero at token budgets where verbatim gain has vanished.
The strongest existing filter is n-gram decontamination of the kind evaluated in Investigating Data Contamination for Pre-training Language Models. In their GPT-2 large setting, ground-truth contamination at 10 repetitions raised MMLU from 22.8% to 24.3%, and n-gram filtering did not reliably remove the inflation. We also run exact-match removal in the style of Infini-gram mini and the METEOR > 0.75 page filter from An Open-Source Data Contamination Report for Large Language Models. Every filter's removed-token fraction is reported, so a filter cannot win by deleting more data. The kernel-chosen edit-distance filter beats these if it brings gain within the clean model's seed variance while removing no more clean tokens than the METEOR filter.
Two ablations isolate the key ingredient. An embedding-similarity filter, removing documents whose sentence embedding sits within a cosine threshold of a test item, with the threshold set to match the removal fraction, tests whether string distance is the right axis at all. A random-radius filter, whose edit radius is drawn without consulting the kernel, tests whether the measured kernel adds value over picking a threshold by hand.
Why not just…?
Why not just rely on exact n-gram or MinHash filtering? It is fast and already standard. But the GPT-2 injection studies report that n-gram filters left contaminated models inflated, and Infini-gram mini states that exact matching cannot see rephrased copies. Our design measures what exact filters miss directly. If the kernel shows $A(d,n)\approx 0$ for every $d$ above the distance that exact matching already catches, this alternative wins and we will say so. That is a useful result, since it would mean fuzzy filtering is wasted compute.
Why not just train on more clean data and let the model forget? This is the most serious threat, because the forgetting results are strong. Two things limit it. The evidence covers only verbatim items. And the re-contamination trend documented in Infini-gram mini means leaks keep arriving throughout training rather than only early. Our extended-training arm settles the question: if $\tau(d)$ is flat in $d$, paraphrased leaks fade as fast as verbatim ones and this alternative is adequate for one-time leaks. We add a spaced-injection arm, spreading the same repetitions across the run, to test the realistic re-contamination case.
Why not just use semantic embeddings instead of edit distance? Edit distance is blind to paraphrase without word overlap, so embeddings could dominate. We treat this as an open empirical question and include it as an ablation at matched removal fraction. If embeddings win at large $d$ while edit distance wins at small $d$, the right filter is a hybrid. The kernel tells us where to switch.
Why not just evaluate on rephrased or perturbed test sets, or at high temperature? These diagnose a finished model but do not tell a corpus builder what to remove. Generalization or Memorization also shows that rephrased leaks still sharpen output distributions, so a rephrased test set may itself be partially contaminated. We evaluate on perturbed GSM8K as a diagnostic, but our question concerns the training side.
Proposed Experiments
The first decisive check is the shape of $A(d,n)$. If initial gain from variants with small $d$ (typos and formatting) is within one standard error of verbatim gain, while full paraphrases give a positive but smaller gain, the core claim is supported. A gain that falls to zero immediately past $d=0$ would refute it.
The second check is the shape of $\tau(d)$. We read it from checkpoints at 1×, 2×, and 5× compute-optimal clean tokens after injection, at each model size. The hypothesis that paraphrased leaks are harder to forget predicts $\tau$ increasing in $d$, plus a residual gain for large-$d$ variants at budgets where verbatim gain has decayed by the roughly 87% reported for OLMo-7B. Number-perturbed GSM8K variants are a control: they should give no transfer to the original answers. If they do, our gain measures format familiarity, not leaked content.
The third check is whether edit distance is the right summary. A model of $G$ that uses only $d$ should fit nearly as well as one with generation-method indicators. A large gap means "distance" must be redefined.
The fourth check is the filter itself. On a corpus seeded with a realistic mixture of distances, the kernel-chosen edit-radius filter should match the clean model's ARC, MMLU, and GSM8K scores within seed variance, at equal or lower removed-token fraction than METEOR, embedding, and random-radius filters. As an external validation, we run the same seed-then-verify search over a FineWeb-Edu shard and report how many additional dirty benchmark items fuzzy matching finds beyond exact matching. Weighting those items by the kernel converts a raw match count into an estimate of expected score inflation.
References
- An Open-Source Data Contamination Report for Large Language Models
- How Much Can We Forget about Data Contamination?
- Quantifying the Effect of Test Set Contamination on Generative Evaluations
- Does Data Contamination Make a Difference? Insights from Intentionally Contaminating Pre-training Data For Language Models
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index
- The Influence of Minimum Edit Distance on Reference Resolution
- Fast Approximate String Matching with Suffix Arrays and A* Parsing
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Measuring Massive Multitask Language Understanding