Live coding and reasoning benchmarks guard against leakage with one rule. A model is scored only on problems published after its training data was collected. That rule treats each model's test set as a fixed, sealed group. In reality a benchmark behaves more like an animal population that researchers keep counting over time: new problems keep arriving, and each problem slowly "dies" as solutions to it spread across the web and into later training corpora. Ecologists handle this with open-population models, which estimate how many individuals are present, how long they survive, and how many new ones join, even though no single survey sees every animal. We propose applying the same approach to benchmarks. Each problem is an individual whose clean-or-exposed status is hidden. Each model release is one survey that observes the problem at a known age. The model estimates how quickly problems become exposed and how much exposure inflates scores.
The payoff is two quantities that current practice cannot provide. The first is a contamination premium that can be measured on the full problem pool, correcting for the fact that newer problems may simply be easier or harder. The second is an empirical shelf life for benchmark items. That shelf life would tell a live benchmark how often it actually needs to replace questions, instead of using a fixed heuristic such as LiveBench's monthly one-sixth rotation.
Core Claim
Our claim is that a staggered-release design removes the difficulty drift that confounds simple pre/post-cutoff comparisons. Every problem is scored by models whose training cutoff predates its release, which are clean controls for that problem, and by models trained after its release, which may be exposed. Placing a latent exposure state on top of this design, with an exposure hazard that depends on problem age and popularity, recovers per-item contamination and its time course. It does so better than cutoff splits and better than probability-based detectors, including when the leaked content has been paraphrased so that string and likelihood signals vanish.
Where This Sits in Prior Work
The motivating evidence comes from longitudinal contamination studies. To the Cutoff... and Beyond? used training cutoffs as a natural experiment on Codeforces and Project Euler. It found that GPT-4's Codeforces pass rate fell from about 37% before the cutoff to 16% after, and that each unit of log GitHub presence raised GPT-4's odds of passing pre-cutoff Project Euler problems by 47.8%, with no effect after the cutoff. That study compares two fixed time bands and cannot separate contamination from a change in difficulty across eras. Test of Time constrains any design that relies on the size of the cutoff "cliff." It showed that perturbing variable names and problem descriptions on LiveCodeBench erased the post-cutoff drop. It also found that influence-function attribution on OLMo2-7B-Instruct fell from 77.5% on cloze questions to 17.5% on transformed questions, so the absence of a cliff does not mean the absence of contamination. Is Your LLM Outdated? shows that post-cutoff text streams are themselves non-stationary; arXiv text became more predictable after March 2024. This is a further reason that raw post-cutoff scores mix contamination with drift.
The benchmarks we build on enforce isolation only by filtering. LiveCodeBench timestamps 612 problems (May 2023 to August 2024) and discards those published before a model's cutoff. LiveBench rotates about one-sixth of its questions each month. Neither measures how long an item stays clean. We reuse both as data sources rather than replacing them.
The method comes from ecology and epidemiology. Jolly-Seber open-population capture-recapture, used with the POPAN parameterization by Belo et al., estimates survival and recruitment from repeated imperfect surveys. That study also shows the analogous failure of one-off interventions: sterilization did not reduce a dog population because outside abandonment kept replenishing it. The benchmark parallel is one-time decontamination while leakage continues. Wilder et al. model flows between compartments of a changing population, which is the structure we borrow for the clean-to-exposed transition. Endogenous Macrodynamics in Algorithmic Recourse is the closest machine-learning precedent for dynamic population effects on models, and it explicitly assumes a closed population. We do not study recourse, but we take its stated limitation seriously in a setting where the population is open by construction.
The Model
Let $y_{im}$ be whether model $m$ solves item $i$. Let $a_{im}$ be the age of item $i$ at model $m$'s training cutoff, which is negative if the item was released after the cutoff. Let $z_{im}\in\{0,1\}$ be the latent indicator that item $i$ was in $m$'s training data. We fit
$$ \Pr(y_{im}=1)=\sigma\big(\theta_m-b_i+\delta\, z_{im}\big),\qquad \Pr(z_{im}=1)=1-\exp\!\Big(-\!\int_0^{\max(a_{im},0)}\!\lambda_0(u)\,e^{\beta^\top x_i}\,du\Big), $$
where the symbols are as follows. $\sigma$ is the logistic function. $\theta_m$ is model ability and $b_i$ is item difficulty. $\delta$ is the contamination premium. $\lambda_0$ is a baseline exposure hazard as a function of item age. $x_i$ holds item covariates such as GitHub presence and platform.
The identifying structure is that $z_{im}=0$ is known whenever $a_{im}<0$. Difficulty $b_i$ is therefore anchored by earlier-cutoff models that are clean for item $i$, and ability $\theta_m$ is anchored by items released after $m$'s cutoff. Only the excess success of later models on older items feeds $\delta$ and $\lambda_0$. That excess is measured after accounting for each model's ability and each item's difficulty, and the difficulty correction is what handles drift.
To handle laundering, each item is also scored in a perturbed form, following the Test of Time transformations. A separate premium $\delta'$ applies to that form, so we can measure how much of the benefit survives paraphrasing. Where logits are available, we add Min-K% Prob scores as a noisy observation of $z_{im}$. This mirrors the imperfect detection that capture-recapture models are designed for. Fitting uses Bayesian inference over the latent $z$. From the fitted model we read off the half-life of cleanliness, which is the age at which $\Pr(z=1)$ reaches one half, and the rotation rate needed to keep the expected exposed fraction below a target.
Research Program
The work has three parts. The first is a controlled ground-truth study in which leakage is planted deliberately. The second is an observational study on real benchmarks and models. The third is a policy analysis that turns the estimated hazard into a replacement rate for live benchmarks.
The controlled study is what gives the proposal teeth, because observational fits alone cannot be checked against truth. Starting from an open-data base model in the OLMo family, we will produce a series of six to eight continued-pretraining checkpoints. Each checkpoint has a successive synthetic cutoff. The inserted data includes LiveCodeBench problems and solutions, each problem included with a probability that rises with its age at that cutoff. This simulates web diffusion under a known hazard. Some insertions are verbatim and some are paraphrased or variable-renamed. Because the true $z_{im}$, $\delta$ and $\lambda_0$ are known, every estimator can be scored exactly. We will also vary hazard shape and difficulty drift. In the drift condition, newer problems are made systematically easier by construction, testing whether the model avoids mistaking drift for contamination.
The observational study fits the model to LiveCodeBench and to the Codeforces and Project Euler archives from Roberts et al. It uses a panel of open and closed models whose cutoffs are published, treating each as one survey. For closed models without logits, the detector term is dropped.
Baselines
The naive reference point is the raw pooled score, with no filtering and no correction. It shows how much ranking and absolute accuracy move once contamination is modeled.
The strongest method used in practice is chronological filtering as in LiveCodeBench. Each model is scored only on items released after its cutoff. We apply it to the same item pool and model panel. Beating it on clean-score estimation has two parts. Our full-pool corrected scores must match filtered scores in the controlled study, which checks that the correction is valid. They must also show lower variance and permit comparisons between models with different cutoffs on a shared item set, which filtering cannot do. Filtering produces no estimate of the premium or of shelf life, so on those outputs the comparison is against the next baseline.
The strongest existing estimator of the contamination premium is the pre/post-cutoff logistic regression with a popularity interaction from To the Cutoff... and Beyond?. Its reported figures are an odds increase of 47.8% per unit log GitHub presence before the cutoff and a Codeforces pass rate falling from about 37% to 16% for GPT-4. We will refit it on identical data. We count it as beaten if, in the controlled study, our premium estimate has lower error than its before/after contrast. The drift condition is where we expect the regression to be biased.
For per-item detection, the baseline is Min-K% Prob as surveyed in A Comprehensive Survey of Contamination Detection Methods. The survey reports that it beats perplexity by up to 10 ROC-AUC points; no reported result was found on Lacuna for it on LiveCodeBench. We run it on the same checkpoints and count it as beaten if our posterior over $z_{im}$ achieves higher ROC-AUC against the planted truth, especially on paraphrased insertions.
The key ablation is item-response modeling with the same staggered anchoring but no latent exposure state or hazard. It is essentially a staggered difference-in-differences. It isolates whether the open-population component adds anything beyond difficulty correction.
Why not just…?
Why not just filter by release date, as LiveCodeBench does? For obtaining one clean score for one model, filtering works, and we do not claim otherwise. It fails at three other jobs. Each newer model is left with fewer items, so its score rests on a smaller set. Filtered scores for models with different cutoffs come from different item sets and cannot be compared directly. Nothing in filtering tells a benchmark maintainer how fast items go stale. Test of Time also shows that a clean-looking post-cutoff window can hide laundered pre-cutoff knowledge. The controlled study measures whether corrected full-pool scores agree with filtered ones, so this alternative is a benchmark for our validity, not something we argue past.
Why not just run the Roberts et al. cutoff regression? It compares eras, and eras differ in difficulty. Test of Time's +2.19-point post-cutoff gain on physics questions and FreshBench's drop in bits per character after March 2024 both suggest that this drift is real. The drift condition of the controlled study tests directly whether the regression attributes planted drift to contamination while the staggered model does not.
Why not just use a likelihood detector such as Min-K% on each item? Closed models expose no token probabilities. Paraphrasing is precisely the case where memorized content no longer shows up in token likelihoods, as the 77.5%-to-17.5% attribution drop illustrates. We keep the detector as one noisy observation inside the model. If it alone matches our AUC on paraphrased insertions, the latent-state model is not needed for detection, and we will report that.
Why not just do difficulty-corrected scoring without the hazard? This is the real threat. If the ablation matches the full model on premium and score error, then for scoring the open-population machinery is decoration. The hazard's contribution would then reduce to the shelf-life estimate. That estimate would still have to be validated by predicting held-out exposure of items at ages not seen during fitting. We will report whichever outcome occurs.
Decisive Checks
The central test is the controlled series. The model succeeds if, across hazard shapes and with and without difficulty drift, it recovers the planted $\delta$ and the half-life of cleanliness within their credible intervals. It must also beat the cutoff regression on premium error and Min-K% on item-level ROC-AUC for paraphrased insertions. If it fails when drift is present, the core claim is wrong.
On real data, the key outputs are the estimated $\delta'$ on perturbed LiveCodeBench items and its size relative to $\delta$. A clearly positive $\delta'$ would be the first quantitative estimate of how much laundered contamination survives, which Test of Time could only establish qualitatively. We will test the fitted hazard prospectively: estimate it on models released up to one date, then predict the excess success of the next released model on older items. From the validated hazard we compute the rotation rate that keeps the expected exposed fraction under 10% and compare it with LiveBench's one-sixth monthly rule. That comparison says whether the heuristic replaces items too often, not often enough, or about right.
References
- To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Is Your LLM Outdated? A Deep Look at Temporal Generalization
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Abundance, survival, recruitment and effectiveness of sterilization of free-roaming dogs: A capture and recapture study in Brazil
- Preventing Infectious Disease in Dynamic Populations Under Uncertainty
- Endogenous Macrodynamics in Algorithmic Recourse
- A Comprehensive Survey of Contamination Detection Methods in Large Language Models