Automated systems now read thousands of AI papers, pull out every reported benchmark score, and pool them into continuously updated databases. These "living" syntheses are used to answer questions such as how much chain-of-thought prompting helps, or how fast models are improving. Every one of these systems takes a reported number at face value. Many of those numbers come from benchmarks whose test questions had already leaked into the models' training data, a problem called benchmark contamination. A living synthesis built on such scores pools memorization together with capability and reports the mix as progress.
This proposal treats the literature's own record of reported scores as a natural experiment for estimating contamination. Each benchmark has a public release date, and each model has a training cutoff. Models whose cutoff falls before a benchmark's release cannot have memorized it. Fresh "live" benchmarks, built from material published after every model's cutoff, give a contamination-free measure of each model's ability. Combining these two facts lets us estimate, for each benchmark, how far reported scores rise above what a model's ability predicts once the benchmark has been public long enough to be scraped. That estimate can then correct the pooled conclusions of a living meta-analysis. The method needs no access to model weights, so it also covers closed models.
Core Claim
Benchmark inflation from contamination can be identified at the level of the whole literature, using only reported numbers, release dates, and training cutoffs. Correcting for it changes some published meta-analytic conclusions in a measurable way. The closest existing system, LLMEvalDB, extracts 18,127 results from 1,737 arXiv papers with 100% precision on model names and metrics. It then pools them uncritically: its headline finding of a +14.61 point chain-of-thought gain on math and symbolic reasoning is computed on benchmarks such as GSM8K and MATH, which are widely believed to be contaminated. Live benchmarks take the opposite approach. LiveBench, LiveXiv and SCALAR escape contamination by discarding old tasks and evaluating on new ones. That choice destroys the fixed, long-running targets a meta-analysis needs.
Our proposal does what neither side does. It keeps the old benchmarks in the synthesis, uses the live benchmarks only as anchors, and estimates the inflation separating the two. This resolves the tension between rotating benchmarks and stable longitudinal pooling: rotating benchmarks become the reference that makes static ones interpretable again.
The Model
Let $y_{p,m,b}$ be the score that paper $p$ reports for model $m$ on benchmark $b$, rescaled to a log-odds scale so that scores near the ceiling are not compressed. We fit the following model:
$$ y_{p,m,b} = \theta_{m,s(b)} - \delta_b + \gamma^\top x_p + \lambda_b \, g(E_{m,b}) + \rho_p + \varepsilon_{p,m,b}. $$
The terms are as follows:
- $\theta_{m,s(b)}$ is model $m$'s ability on the skill category $s(b)$ that the benchmark tests.
- $\delta_b$ is the benchmark's difficulty.
- $x_p$ encodes the prompting setup in the paper, such as chain-of-thought or number of demonstrations. $\gamma$ holds the method effects a meta-analysis wants to estimate.
- $E_{m,b}$ is how long benchmark $b$ had been public before model $m$'s training cutoff. It is set to zero if the cutoff came before the release.
- $g$ is a saturating function of that exposure, since the number of web copies of a benchmark grows and then plateaus.
- $\lambda_b \ge 0$ is the per-benchmark inflation we want to estimate.
- $\rho_p$ absorbs paper-level reporting bias, for example self-reported results versus third-party reproductions.
- $\varepsilon_{p,m,b}$ is residual noise.
Identification comes from three sources of variation. First, live-benchmark scores have $E=0$ by construction, which pins down $\theta$. Second, models trained before $b$ was released pin down $\delta_b$. Third, the same model reported on old and new benchmarks within one skill separates inflation from the obvious confound that newer models are both better and more exposed. This design resembles a difference-in-differences comparison across release dates and cutoffs.
We will add an interaction between $x_p$ and $E_{m,b}$. This tests whether method gains themselves are distorted on contaminated benchmarks. One plausible mechanism is that a model which has memorized answers gains less from reasoning prompts.
Research Program
The first component is the data substrate. We extend the LLMEvalDB pipeline, which reads LaTeX source from arXiv and extracts evaluation tables, to papers through 2025. We add three fields that the pipeline currently discards: the paper date, whether the evaluation was run by the model's creators, and the exact benchmark version.
Model training cutoffs come from model cards. Where a cutoff is undisclosed, we treat it as a latent variable with a prior bounded by the release date. Benchmark release dates come from the first arXiv version of the benchmark paper. Fresh anchors come from published LiveBench, LiveXiv and SCALAR leaderboards, which share many models with the extracted literature.
The second component is the model fit. We fit it with hierarchical Bayesian inference so that benchmarks with few reports borrow strength from others in the same skill. This gives a posterior over $\lambda_b$ for every benchmark with enough coverage.
The third component is the living layer. Each meta-analytic conclusion in the database, such as "chain-of-thought helps math by about 15 points," is stored as a claim with a state. Following the deterministic state policies of ClaimGarden, a claim moves from supported to qualified when its contamination-corrected interval excludes the uncorrected estimate. ClaimGarden itself was shown only on a three-claim smoke test and assumes its upstream evidence is clean. Here the claims number in the hundreds, and the upstream evidence is precisely what is suspect.
Baselines
The naive reference point is uncorrected pooling, which is the current practice of LLMEvalDB: average reported gains across papers, weighting every number equally. We reproduce its published analyses on its own records, including the +14.61 chain-of-thought gain and the +3.0 median few-shot gain, and then rerun them on our extended data. Any correction must be compared against this.
The strongest simple competitor is a cutoff filter. It keeps only results where the model's cutoff predates the benchmark's release, and treats everything else as untrustworthy. This filter is clean, but it throws away most of the data on popular benchmarks. Beating it means producing method-effect estimates that agree with the filter where the filter has enough data, while giving tighter intervals and better out-of-sample predictions where it does not.
The strongest instance-level detector in the file is Min-K% Prob membership inference, which Unveiling Causal Reasoning in Large Language Models used to confirm that CausalProbe-2024 was fresh. No reported result was found on Lacuna for its accuracy in detecting benchmark-level contamination. We therefore run it ourselves on the same open-weight models and benchmarks used in validation, with matched compute. Beating it means that our literature-derived $\lambda_b$ ranks benchmarks by measured inflation at least as well as Min-K% scores do, while also covering closed models, which Min-K% cannot.
The key ablation removes the live-benchmark anchors and estimates $\theta$ only from the static benchmarks themselves. If the anchors matter, this version should confound ability with inflation and track measured inflation worse. A second ablation drops the exposure term entirely, leaving a standard item-response-style difficulty model like the one LiveXiv uses to cut evaluation cost. That isolates whether release dates add information beyond difficulty.
The strongest measured benchmark numbers in the file are LiveBench's top overall score of 64.7% for o1-preview and the SciFact-Open drop of MultiVerS from 72.5 to 52.4 F1. Neither is a direct competitor score for contamination estimation. No reported result was found on Lacuna for contamination-corrected meta-analysis.
Why not just…?
Why not just apply the cutoff filter and stop? This is the most serious alternative, and it may suffice for heavily studied skills where many pre-release models exist. Its weakness is that benchmarks like GSM8K and MATH were released before most current models, so the filter leaves conclusions about current models almost entirely unsupported. Our experiments report the filter's estimates beside ours. If the two agree everywhere the filter has coverage, and our extra coverage does not improve held-out prediction, the right conclusion is that the filter is enough. In that case we would publish the extended, date-annotated database as the main contribution.
Why not just run membership inference on every benchmark? Methods like Min-K% Prob need token probabilities, so they cannot audit closed models. Closed models are exactly the ones most often reported and least often verified. Min-K% also signals whether text was seen, not how much it inflates the score. The validation experiment measures how well Min-K% scores correlate with actual score inflation on open models, which tells us whether it can stand in for our estimate where it is available.
Why not just abandon static benchmarks and use live ones? Live benchmarks answer "which model is best now," but the meta-analytic questions concern method effects across thousands of datasets. LLMEvalDB covers 2,984 of them. LiveBench refreshes about one sixth of its questions monthly across six categories and cannot reproduce that breadth. The two also answer different questions, and the LiveXiv finding that rankings shift on fresh data shows they do not simply agree with static results.
Why not just re-evaluate the models yourself on rephrased test sets? This is the gold standard, and we use it for validation. It is too costly for the whole literature, and it is impossible for discontinued closed models. ScienceAgentBench's manual, one-time data perturbation illustrates the effort involved.
Proposed Experiments
The decisive validation constructs measured inflation for open-weight models in the Llama and Qwen families. We evaluate each model on original items and on rephrased or numerically perturbed versions of roughly fifteen popular static benchmarks, in the spirit of ScienceAgentBench's perturbation strategy. The gap between original and perturbed scores is the measured inflation. The primary metric is the Spearman correlation between the literature-estimated $\lambda_b$ and this measured gap across benchmarks. Min-K% scores and the two ablations are compared on the same metric.
A second check is forecasting. We fit the model on literature available up to a given month, then predict models' scores on LiveBench questions released afterward. Prediction error is compared against predictions from raw reported scores and from cutoff-filtered scores.
A third check uses known structure in prior results. The 2022 versus 2025 gap in SCALAR and the ranking drops in LiveXiv should be partly reproduced by our estimated inflation for the corresponding models.
Finally, we re-estimate the chain-of-thought and few-shot effects reported by LLMEvalDB with and without correction, and test the interaction between method and exposure. A corrected chain-of-thought gain whose interval excludes +14.61 would be the headline result. A corrected estimate that matches the original is also informative, because it would show that this particular meta-analytic conclusion holds up under contamination, and the living layer would record it as supported.
References
- Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- LiveXiv - A Multi-Modal live benchmark based on Arxiv papers content
- SCALAR: Scientific Citation-based Live Assessment of Long-context Academic Reasoning
- ClaimGarden: Update-Aware Claim-State Control for AI Scientist Workflows
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery