Most benchmark questions probably do not reach a language model's training data as verbatim copies. They reach it second-hand. A tutor posts a GSM8K word problem with a worked solution, a forum user reposts it with the numbers slightly changed, a study-group account turns an MMLU question into a quiz meme, and a blogger translates a HumanEval task and explains it. Communication research calls this pattern two-step flow: a central source reaches intermediaries, who rewrite the message for their own audiences and pass it on. That literature finds the second step often carries more reach than the original. This proposal treats benchmark leakage the same way. We trace benchmark items through their intermediary rewrites in a public pretraining crawl, then measure in controlled training runs how much score inflation each kind of rewrite causes and whether current detectors notice it.
The question matters because decontamination practice is built around the first step. N-gram filters and near-duplicate removal catch mirrors of the source. They are not designed to catch the rephrased, explained, reformatted derivatives that intermediaries produce, and the contamination literature already agrees that this "implicit" contamination is common and evades those filters (survey by Cheng Xu et al.). What nobody has measured is the realistic distribution of such derivatives on the web. It is also unknown whether they inflate scores as much as verbatim copies do, and whether a detector tuned on synthetic paraphrases generalizes to them.
Core Claim
We claim that implicit contamination has cascade structure. The derivatives of a benchmark item differ systematically by intermediary type: answer-bearing tutorials, question-only reposts, translations, and meme or quiz reformattings. They also differ by hop distance from the source. We hypothesize three consequences. First, the inflation an item receives depends on the number and diversity of answer-bearing derivatives, not on raw token count. Second, detector recall falls with hop distance. Third, synthetic LLM paraphrases, the standard stand-in for implicit contamination in existing work, give a biased estimate of both effects.
The third hypothesis is the one that could fail. If real derivatives behave like LLM paraphrases, the practical conclusion is that synthetic paraphrase is a sufficient proxy. That is still worth establishing, but it would retire the social-propagation framing as a design principle.
The communication literature supplies the mechanism and two testable analogies rather than evidence about models. Peng et al. find that secondary dissemination accounts for over 43% of total digital impact, so we expect derivatives to outnumber mirrors in a crawl. Rong Wang describes intermediaries "de-professionalizing" content into community idiom, which predicts large surface drift. Li and Zhao show that viral copy-pasta spreads because users can personalize it, yet sharing has no significant direct effect on purchase intention. The contamination analogue is that exposure is not memorization: some derivative types should "convert" into inflated scores and others should not. We test which ones do.
Relation to Prior Work
The closest work falls into three groups.
Controlled contamination injection. DCR defines four levels of contamination (semantic, information, data, label) and injects each artificially. CDD/TED detects explicit and rephrased contamination from how peaked a model's sampled outputs are. Both use synthetic contamination. We instead draw the contaminating text from real web derivatives and compare it against their synthetic forms.
Contamination in socially spread content. TripleFact and SSA show that fake-news detection scores are inflated by semantically related material seen in pretraining. They treat that material as a static corpus and do not model how it was produced or who rewrote it.
Dynamic benchmarks. EvoEval, DyVal and P⁴Bench avoid contamination by generating new or hidden items. This motivates our work: EvoEval's 38.1% average pass@1 drop shows how much static scores rely on familiar surface forms. It does not tell a practitioner auditing a static benchmark which web text to remove.
FineWeb supplies our substrate. It documents a 15-trillion-token crawl with per-snapshot MinHash deduplication, which is exactly the filter whose blind spot we want to measure.
Research Program
The first component is a derivative-tracing index over FineWeb for the test sets of GSM8K, ARC-Challenge, MMLU and HumanEval. Candidate derivatives are retrieved by two routes. The first is dense-embedding similarity to the item. The second is a task-specific "solution signature" that survives rewording: the numeric answer plus the set of operands for GSM8K, the answer option text for ARC and MMLU, and the function signature plus the behavior of the reference tests for HumanEval.
A stratified sample of candidates is then hand-labeled on three attributes:
- Real derivative or coincidental match.
- Intermediary type: mirror, answer-bearing explanation, question-only repost, translation, or reformatting such as a quiz, meme, or flashcard.
- Whether the answer is present.
Those labels train a classifier that scales labeling to the full candidate set. Because FineWeb spans 96 Common Crawl snapshots, each item's derivatives can be ordered by first appearance. Combined with textual lineage (which derivative shares the most specific phrasing with which earlier one), this gives an approximate hop distance for each derivative.
The second component is a controlled training study, which is needed because we cannot observe what frontier models trained on. We take small open base models (Qwen2.5-1.5B and a FineWeb-trained ablation-scale model) and continue pretraining them on clean FineWeb text with benchmark items injected at matched token doses. Four injection conditions are compared:
- Verbatim copies.
- LLM paraphrases, following the synthetic rephrasing used by CDD and DCR.
- Real traced derivatives, sampled to match intermediary-type proportions.
- Simulated cascades, produced by chaining persona-conditioned LLM rewrites (tutor, then forum reposter, then quiz account) up to three hops.
Within each item we randomize whether it is injected and in which form, so that item difficulty is held constant. Inflation is accuracy on exposed items minus accuracy on matched unexposed items. We additionally test whether inflation carries over to the rephrased HumanEval_R and EvoEval variants, which separates familiarity with surface form from memorized answers.
Baselines
Do nothing. The naive reference is the raw benchmark score with no decontamination and no adjustment. For the index, the naive filter is n-gram overlap decontamination of the kind described in the contamination survey; no reported result found on Lacuna for its recall against real web derivatives. FineWeb's per-snapshot MinHash deduplication is a second pipeline baseline, since it is what a practitioner already runs; no reported result found on Lacuna for its recall on benchmark derivatives. Both are run over identical candidate pools and scored against the same hand labels. Beating them means our index recovers substantially more labeled derivatives at matched precision.
Output-distribution detection (CDD). CDD samples many generations per prompt and flags items where the samples are nearly identical to one another, a sign of memorization. It is the strongest white-box-free detector that explicitly targets rephrased contamination. It reports relative gains of 21.8% to 30.2% in accuracy and AUC over n-gram and LLM-based detectors on DETCON (Dong et al.). We run it on the injected models with its published sampling budget in every condition.
Reference-model difficulty correction (ConStat). ConStat compares a model's score with the score predicted from reference models evaluated on rephrased benchmarks. It reports detecting 98.4% of sample-specific contamination where baselines reached at most 71%, which makes it the most serious threat. Its reference panel is the same set of uninjected checkpoints for every condition. Our claim is beaten if ConStat's detection power on real-derivative and cascade injections matches its power on paraphrase injections at equal dose. That result would mean the social framing adds nothing to detection.
Level-based adjustment (DCR). DCR serves as the score-adjustment baseline. It reports adjusted accuracies within 4% of uncontaminated baselines. We ask whether that holds when the contamination is real derivatives rather than its own injected levels.
Key ablation. The ablation that isolates our ingredient holds total injected tokens fixed while varying the number of distinct intermediary derivatives, and separately varies answer presence. If inflation tracks tokens alone, the "diversity of intermediaries" claim fails.
Why not just…?
Why not just generate paraphrases with an LLM and call that implicit contamination? This is the most serious alternative, and our design is built to settle it rather than argue with it. Existing detectors were validated this way (Dong et al.; DCR). The paraphrase condition sits alongside the real-derivative condition at equal dose. If inflation and detector recall do not differ between the two, we will report that synthetic paraphrase is an adequate proxy and drop the cascade model. The evidence for a difference is only indirect. EvoEval shows that the kind of transformation matters: drops of 74.7% on Combine versus about 4% on Concise (EvoEval). Real intermediaries plausibly add solutions, translations and format changes that an instruction to "paraphrase" does not.
Why not just run embedding-based semantic decontamination against the benchmark? It may work well, and it is one of our retrieval routes. The open question is its recall on translated and answer-bearing derivatives far from the source, which have low embedding similarity to the item. We report recall stratified by hop distance and intermediary type, so if embedding filtering alone saturates recall, that will be visible directly.
Why not just abandon static benchmarks for dynamic or cryptographic ones? DyVal and P⁴Bench are confined to formal domains such as graphs, SAT and discrete logarithms. Static benchmarks like MMLU and GSM8K remain the headline numbers people report. Any published dynamic item also becomes a source node for new intermediaries, so the problem recurs.
Proposed Experiments and Decisive Checks
The descriptive experiment reports, per benchmark:
- the ratio of derivatives to mirrors;
- the mix of intermediary types;
- the fraction of derivatives that survive n-gram filtering and MinHash deduplication, by hop distance.
The first decisive check is the dose-diversity ablation. Our claim predicts that, at fixed tokens, more distinct answer-bearing derivatives produce more inflation, while question-only derivatives produce little. This is the Li and Zhao "sharing is not conversion" pattern carried over to models.
The second decisive check is whether detection AUC for CDD and ConStat falls from the verbatim condition, through the paraphrase condition, to the real-derivative condition. A drop in the last step is the finding that would change decontamination practice.
The third check is external validity. We ask whether the simulated-cascade generator reproduces the inflation and detectability of real derivatives. If it does, auditors can stress-test detectors without crawling the web.
The main risks are three. Continued pretraining at small scale may understate memorization. GSM8K accuracy may be too low at 1.5B parameters, which is why ARC and MMLU are included. Hop distances estimated from snapshot order are noisy, so we report the hop analysis only on hand-verified lineages.
References
- Benchmark Data Contamination of Large Language Models: A Survey
- Cultural transmission ability of urban heritage along the Jiangsu section of the Beijing–Hangzhou Grand Canal
- Study of Advertising Communication Form in the New Media Age
- The Study of Development In ‘Joke’ Marketing: From the Point of KFC ‘Crazy Thursday’
- DCR: Quantifying Data Contamination in LLMs Evaluation
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models
- TripleFact: Defending Data Contamination in the Evaluation of LLM-driven Fake News Detection
- SSA: Semantic Contamination of LLM-Driven Fake News Detection
- Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- ConStat: Performance-Based Contamination Detection in Large Language Models