Benchmark contamination is usually measured as a single number per test item: how often it appears on the web, or whether it was published before or after a model's training cutoff. The web does not leak a benchmark item once, though. A contest problem first shows up as a statement. Then solution write-ups and forum threads arrive in a rush around the contest date, and afterward some copies persist in archives and tutorial repositories for years. We propose to measure contamination as a time series of exposure across the dated snapshots of a public web crawl, using the burstiness statistics that information retrieval developed for event search. The test is whether the shape of that exposure predicts what a model can do, beyond its total volume. Shape here means a short burst of paraphrases versus steady verbatim persistence across many crawl waves.
The reason to expect shape to matter is mechanical. Modern pretraining corpora are deduplicated within each crawl snapshot, not across all of them. FineWeb found that per-snapshot deduplication beats global deduplication over its 96 Common Crawl snapshots. Under that pipeline, an item copied verbatim 1,000 times in one month survives as roughly one copy. An item that sits on the same page across 40 snapshots survives as 40 copies. A burst of non-identical paraphrases, such as forum explanations and blog solutions, survives almost entirely. Static counts treat all three cases the same. We expect them to produce different kinds of contamination: verbatim persistence should drive literal recall, and paraphrase or concept bursts should drive success on rephrased versions. If that holds, it would explain the split Test of Time observed between cloze questions and transformed questions.
Core Claim
For each benchmark item, we decompose its exposure in a time-stamped crawl into three channels: verbatim copies, paraphrased restatements (solutions, explanations), and bursts of the item's underlying concept vocabulary. Within each channel we summarize the time series by total volume, persistence across snapshots, and burstiness. We claim that, after per-snapshot deduplication, persistence of verbatim copies predicts cloze-style recall, and that pre-cutoff paraphrase and concept bursts predict performance on semantically transformed items. Both effects should hold over and above total count. We will establish this observationally on open crawl data and causally by injecting items with controlled temporal profiles into small pretraining runs.
How This Relates to Prior Work
Two contamination papers motivate the gap. To the Cutoff... and Beyond? uses training cutoffs as a natural experiment on Codeforces and Project Euler. It regresses pass rate on a static "GitHub Presence" count and finds that each unit of log presence raised GPT-4's pre-cutoff odds of passing a Project Euler problem by 47.8%, with no effect after the cutoff. That metric treats exposure as time-invariant; we replace it with a profile over time. Test of Time builds parallel cloze and LLM-transformed questions from 20,277 arXiv papers. Cloze questions show a post-cutoff drop (−3.83 points for GPT-4o-mini), while transformed questions show none and even a +2.19-point post-cutoff rise in physics. Influence-function traceability falls from 77.5% to 17.5% under transformation. The authors attribute the rise loosely to "temporal variation" but have no tool to measure it. Our concept-burst channel is designed to test that explanation directly.
Two other papers constrain the design. Rethinking Benchmark and Contamination with Rephrased Samples shows that rephrased test items evade n-gram filters yet lift Llama-2-13B on MMLU from about 55 to about 86, and that an LLM-based decontaminator detects such rephrasings with F1 of 0.94–1.0. Counting only verbatim matches is therefore not enough, and that decontaminator is a usable matcher for our paraphrase channel. FineWeb's deduplication finding supplies the mechanism that makes persistence and burst count differently. The contamination survey places our work among comparison-based methods of the chronological-analysis kind; our contribution is to refine "chronological" from a cutoff indicator into an exposure profile.
The method comes from temporal retrieval. Xu et al. describe "evergreen bias," where steady background terms swamp transient events. Van Canneyt et al. score terms by focal-window frequency relative to a historical baseline. Zhou et al. use the coefficient of variation across time windows. DynaMiTE formalizes Burst Inverse Time Frequency (BITF), a score for terms that spike in one window while staying rare before and after. Structured Event Retrieval over Microblog Archives shows that scoring by burst profile beats co-occurrence feedback for event retrieval, with P@10 of 61% versus a keyword baseline. These papers retrieve events from streams. We repurpose their statistics to describe how a benchmark item, treated as an "event," was spread across the crawl.
The obvious intersection, using temporal query expansion to stop search agents from retrieving leaked answer keys as studied in Search-Time Data Contamination, is deliberately not our focus. In that setting the evaluator already holds the benchmark and can simply filter retrieved pages that contain the question or answer. That simple check would likely match any burst-based filter.
Research Program
We first build exposure series. For item $i$ and crawl snapshot $s$, let $v_i(s)$ count documents containing a verbatim match (long n-gram overlap) and $p_i(s)$ count documents the LLM decontaminator judges to be a restatement or solution. These two counts use a candidate set retrieved by BM25 and dense search over the snapshot. Let $k_i(s)$ be the concept channel, computed in two steps. We first extract the item's technical terms, such as a theorem's named objects or a problem's key algorithmic phrase. We then measure their BITF-style spike in snapshot $s$ against neighbouring snapshots. For each channel $x$ we compute total volume $\sum_s x_i(s)$ and persistence (the number of snapshots with $x_i(s)>0$). We also compute burstiness as the coefficient of variation
$$\mathrm{CV}_i = \frac{\sigma_s\left[x_i(s)\right]}{\mu_s\left[x_i(s)\right]},$$
where $\mu_s$ and $\sigma_s$ are the mean and standard deviation over snapshots, following Zhou et al. Finally we compute lead time, the gap between the burst peak and the model's cutoff. Bin width is a known weakness of burst statistics. Common Crawl snapshots fix a natural bin, and we will report sensitivity when adjacent snapshots are merged.
The item pools are Codeforces and Project Euler problems as used by Roberts et al., LiveCodeBench problems (May 2023–August 2024) from LiveCodeBench, and the Test of Time arXiv cloze and transformed questions. Exposure is computed over FineWeb's snapshots, which serve as a public proxy for the web, much as Roberts et al. used GitHub.
For the observational analysis, we fit per-item logistic models of correctness. The predictors are difficulty, the cutoff indicator, and the exposure features, with interactions with cutoff, mirroring Roberts et al.'s design. We fit them separately for cloze and transformed formats. Models include the closed models those papers already evaluated, small models we pretrain on FineWeb snapshots (so exposure is measured on the true training data), and OLMo2-7B as used in Test of Time. For OLMo2-7B we compute exposure on its released training data where snapshot provenance is available, and otherwise mark it as proxy-only.
For the interventional analysis, we pretrain models of a few hundred million parameters for the main grid and confirm at the 1.7B scale FineWeb used for ablations. Before running FineWeb's per-snapshot deduplication, we inject held-out problems, drawn from post-cutoff LiveCodeBench and arXiv items absent from FineWeb, into the snapshot mix. Each problem gets one of several profiles at matched raw count: a single-snapshot verbatim burst, verbatim persistence across many snapshots, a single-snapshot paraphrase burst, and a concept-only burst where related material appears but the item itself does not. We then test each model on cloze and transformed versions of every injected item.
Baselines
The naive reference point is the pre/post-cutoff indicator alone, the standard chronological probe used by LiveCodeBench and Roberts et al. Our features must explain variance among pre-cutoff items, where this indicator is constant, and must account for the transformed-question pattern, where Test of Time shows the indicator fails.
The strongest existing method is Roberts et al.'s static presence regression. Its reported effect is a 47.8% increase in GPT-4's pre-cutoff pass odds per unit of log GitHub Presence on Project Euler, falling to an odds ratio of about 1.0 after the cutoff. We run it on identical items and model outputs in two versions, with GitHub counts and with total FineWeb verbatim count. No reported predictive accuracy (AUC or held-out likelihood) for this metric was found on Lacuna, so we will compute it ourselves under cross-validation with the same folds and the same number of free parameters tuned for every model. Beating it means a statistically significant gain in held-out log-likelihood and AUC from adding temporal-shape features to the count-only model, replicated on at least two item pools.
A semantic-count baseline sums the LLM-decontaminator paraphrase matches over all snapshots and ignores time. It isolates whether gains come from detecting paraphrases at all or from their temporal arrangement.
The key ablation is the injection grid at matched raw count. If the deduplication-aware account is right, verbatim persistence beats the single verbatim burst on cloze accuracy. The paraphrase burst should beat both on transformed accuracy. Rerunning the grid with global deduplication should change which profile wins in the predicted direction, and that is the test of the mechanism.
Why not just…?
Why not just count total occurrences, as GitHub Presence does? This is the real threat. Exposure volume and persistence are correlated, and count may carry most of the signal. The nested-model comparison quantifies how much shape adds beyond count. The matched-count injection grid settles it causally. If the profiles give indistinguishable accuracy at equal raw count, the claim fails, and we will report count as sufficient.
Why not just compare pre- and post-cutoff accuracy? Test of Time shows this signal disappears under question transformation and under variable renaming on LiveCodeBench. A flat curve therefore cannot certify a model as clean, and a cutoff indicator says nothing about variation among pre-cutoff items. Our concept channel also tests a concrete explanation for the +2.19-point physics rise: that post-cutoff papers concern ideas already bursting before the cutoff. That would show up as positive pre-cutoff $k_i$ bursts predicting transformed-question success on post-cutoff items.
Why not just run an n-gram or LLM decontaminator and flag leaked items? Flagging gives a binary answer about presence, and Yang et al. show n-gram flags miss rephrasings. We use their LLM decontaminator as our paraphrase matcher. A flag still cannot say how much an item's exposure helps a model, or why transformed versions remain solvable. The semantic-count baseline measures exactly what flagging plus counting achieves without time.
Decisive Checks
Three outcomes would decide the proposal. In the injection experiment, at matched raw count, cloze accuracy must be ordered by verbatim persistence and transformed accuracy by paraphrase or concept burst, and the ordering must shift when global deduplication replaces per-snapshot deduplication. Observationally, temporal-shape features must improve held-out prediction of pre-cutoff correctness over the static-presence regression on Codeforces or Project Euler and on the arXiv items. Concept-burst lead time must also predict the post-cutoff transformed-question rise in physics. If the first check fails, the deduplication mechanism is wrong. If only the second fails, crawl-based temporal profiles are too noisy a proxy for closed-model training data. Either result tells benchmark builders whether "how often" is enough or whether they must also track "when and in what form."
References
- The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- Benchmark Data Contamination of Large Language Models: A Survey
- ICTNET at Blog Track TREC 2009
- Detecting Newsworthy Topics in Twitter
- WIA-Opinmine System in NTCIR-8 MOAT Evaluation
- DynaMiTE: Discovering Explosive Topic Evolutions with User Guidance
- Structured Event Retrieval over Microblog Archives
- Search-Time Data Contamination
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code