A benchmark is a reserve that gets used up. Each time its questions are published, served to a closed model behind an API, or scraped into a web corpus, some of them stop measuring reasoning and start measuring recall. Benchmark maintainers already act on this. LiveBench, for example, replaces about a sixth of its questions every month. That schedule is set by the calendar, not by how much contamination has actually built up. Actuarial mathematics has studied a problem with the same shape for decades: a firm wants to pay out as much as it can to shareholders without letting its cash reserve hit zero. In that literature, the best payout and refill rules depend on the current reserve level, not on fixed dates.
This note proposes treating a benchmark's uncontaminated reserve the same way. The hard part is not the control mathematics. It is that nobody can directly observe how much uncontaminated reserve is left, and the standard ways of measuring it break down on modern reasoning models. The core contribution is a contamination estimator built from procedurally generated "twin" items that are matched in difficulty to the originals. The estimator works from performance gaps, not from token probabilities, so it should survive the reinforcement-learning step that blinds current detectors. A state-dependent refresh rule is then driven by this estimate and validated against calendar refresh and against simulation-tuned thresholds.
Core Claim
Two claims are being tested, and they can fail independently. The first is measurement. Take an item and generate isomorphic twins at the same structural difficulty, the way DyVal builds reasoning tasks from directed acyclic graphs (graphs of computation steps) with controlled depth and width. A model's accuracy on the item, minus its accuracy on those twins, then estimates how much of that item's score comes from memorization.
The key evidence that this should work comes from On The Fragility of Benchmark Contamination Detection in Reasoning Models. That paper shows that training with Group Relative Policy Optimization (GRPO, a reinforcement-learning method for reasoning models) shrinks the log-probability gap between seen and unseen problems, while the inflated task performance stays. Detectors that read token probabilities, such as Min-K% and LiRA, lose that signal. A detector that reads performance directly should keep it.
The second claim is control. Once the reserve can be estimated at audit times, a refresh rule of the form "when the estimated reserve falls to a lower level, generate a batch that brings it back up to a target level" should keep leaderboard rankings faithful for longer at equal generation cost than a fixed calendar does. This rule shape comes from the fixed-cost band policies proved optimal in Optimal dividends in the dual model under transaction costs.
Formulation
Let $X_t$ be the effective number of uncontaminated items at time $t$. The maintainer chooses three things: $L_t$, the cumulative evaluation use (items served or released); the refresh times $T_k$; and the batch sizes $\xi_k$. The maintainer then maximizes
$$V(x)=\sup\ \mathbb{E}_x\Big[\int_0^{\tau} e^{-qt}\,dL_t-\sum_{k:\,T_k<\tau} e^{-qT_k}\,(K+c\,\xi_k)-\phi\,e^{-q\tau}\Big].$$
The symbols are as follows. $q$ is a discount rate. $K$ is the fixed cost of running any refresh, which covers verification, review and release engineering. $c$ is the per-item generation cost. $\phi$ is the penalty for ruin. $\tau$ is the first time the benchmark's ranking of models drifts too far from a ranking computed on fresh items.
The reserve drains in two ways. Using it drains it, because serving an item to a closed model exposes it to that model's provider, so payouts consume the reserve. It also drains through an exogenous leakage process. Whether that leakage is a smooth drift or a series of jumps at crawl snapshots and model releases is an open empirical question, and the answer decides which result from the stochastic-control literature applies.
If leakage behaves like steady drain, the reserve looks like the dual model studied in ON OPTIMAL DIVIDENDS IN THE DUAL MODEL, and band or barrier rules follow. If leakage arrives in downward jumps, the bail-out and double-barrier results become the relevant reference. Those are On the Bail-Out Optimal Dividend Problem and Optimal dividends and capital injection, the latter of which also handles regime switching. Because the reserve is only seen at audits, the maintainer acts only at discrete review times, the setting where periodic barrier strategies are optimal.
None of these papers deals with anything other than money. Here they supply the structure of candidate policies and the comparative statics, meaning how the thresholds should move as costs change. They do not supply new theorems for this setting.
Research Program
The first component is the twin-gap estimator. For each item in a procedurally generated pool, I generate $m$ twins that share its graph structure, depth and width but differ in surface content and answer. I then estimate per-item inflation as the item's accuracy minus the twin accuracy, pooled with a hierarchical shrinkage model.
The estimator borrows two ideas. From ConStat it borrows the idea of defining contamination as performance that does not generalize. ConStat corrects for item hardness by fitting a function across reference models and a reference benchmark. The twin construction instead removes the hardness difference by design, item by item, with no reference models. From the adaptive-testing work it borrows item response theory, a statistical model that assigns each item a difficulty and each model an ability. Here it is used to flag response patterns where a model succeeds on hard items while failing on easy ones, which is a second signal of memorization. That work assumes a fixed item bank. This proposal tracks how the bank degrades over time.
A real confound exists. If a model learns a template from leaked items, its twin accuracy rises too. The twin gap then underestimates contamination, while the model's "skill" on the template partly reflects the leak. I will measure this transfer directly in the controlled experiments below rather than assume it away.
The second component is a decay measurement. Using LiveBench's dated monthly question releases (LiveBench), I compare two things for each release cohort of questions: how models released after that cohort perform on it, and how the same models perform on cohorts that are still fresh. I then test whether the growing gap is smooth or jumps at model-release dates. The validity-decay theory in AI Benchmark Half-Life in Recursive Corpora predicts decay but does not fit a stochastic process to it or derive a policy. This step supplies the leakage parameters the control problem needs.
The third component is the policy itself, in two versions: a band rule computed from the fitted dynamics, and a periodic-review version that acts only at audit times. Both are evaluated in a simulator calibrated to the decay measurement and in a retrospective replay of LiveBench history.
Baselines
The naive reference point is doing nothing: a static pool that is never refreshed, as with MMLU, which an expert-level benchmark paper notes now exceeds 90% accuracy. It runs on the same initial pool and the same evaluation traffic. Beating it should be easy. Its job is to calibrate how fast ruin arrives with no intervention.
The deployed baseline is LiveBench-style calendar refresh, which replaces one sixth of the items monthly. For a fair comparison, it gets exactly the same total generation budget, meaning the same number of new items and the same number of refresh events, as the proposed rule. I found no reported result on Lacuna for rank fidelity or cost under calendar refresh, so this number will be produced here. Beating it means matching its time-to-ruin with at least a fifth less spending, or extending time-to-ruin at equal spending, across a majority of the calibrated leakage regimes.
The strongest competitor on the estimation side is ConStat, run on the same models and items with its own reference models. Its reported numbers are detection of 89% of syntax-contaminated and 98% of sample-contaminated models (ConStat). Those numbers do not cover models trained with reinforcement learning after contamination. For log-probability detectors, the reference figures are LiRA at 89.1% AUROC (area under the ROC curve, where 50% is chance) before reinforcement learning, falling to 50–58% after it (Fragility). The twin gap beats these if, on models that underwent GRPO, it holds AUROC well above that 50–58% band and matches or exceeds ConStat's AUROC on the same models.
The strongest competitor on the control side is a threshold rule tuned by grid search in the same simulator: "refresh when ConStat's inflation estimate exceeds a threshold, up to a target size." It gets the same tuning budget in simulator rollouts as the band rule gets for parameter fitting.
Two ablations isolate the key ingredients. In the first, the band rule is driven by Min-K%/LiRA estimates instead of the twin gap; this isolates the estimator. In the second, the reserve is observed continuously instead of only at audits; this isolates the cost of discrete observation. An oracle (s,S) rule, a standard inventory policy that sees the true contamination state, sets the upper bound.
Why not just…?
Why not just keep a secret holdout and never release anything? Secrecy addresses public scraping, and P⁴Bench adds publicly verifiable scoring on private suites. It does not stop saturation, and wherever closed models are evaluated through provider APIs, each evaluation exposes items to the provider. That is exactly the usage-linked drain this formulation models. In this framework a secret holdout is the regime with low leakage and few releases. The simulator reports its time-to-ruin from saturation alone, so whether secrecy alone suffices becomes a measured outcome rather than an argument.
Why not just refresh on a fixed calendar? It is simple and already deployed. The fixed-cost result in the transaction-cost dual model suggests it wastes effort when leakage is slow and lets the benchmark fail silently when leakage speeds up. That is a statement about money in a stylized model, so the equal-budget backtest is the actual test. If calendar refresh matches the band rule under realistic, near-stationary leakage, I will report that state-dependence buys nothing at current leakage rates.
Why not just tune a refresh threshold by simulation or reinforcement learning, without the Lévy-process theory? This is a real threat. If the right policy family is a band rule, grid search over two thresholds will find it. The proposal therefore does not rest on the closed form. The theory contributes the policy family and fast recomputation when costs or leakage rates change. The contribution that grid search cannot replace is the state estimate, because no tuning helps if the input signal is noise after reinforcement learning. The baseline comparison will show whether the tuned ConStat threshold and the band rule perform the same. If they do, the claim narrows to the estimator and the decay measurement.
Why not just use better membership-inference detectors? Fragility evaluates ten detection methods on contaminated reasoning traces distilled into models after reinforcement learning, and all of them average 50–58% AUROC. The likely reason is that reinforcement learning removes the log-probability signal these detectors rely on while leaving the accuracy inflation in place. The first ablation measures how much of the control gain disappears when such a detector drives the policy.
Proposed Experiments
The decisive first check uses dose-controlled contamination. I would take DyVal pools at depths D1–D4 and AIME/AMC-style problems, and contaminate 8B-class open reasoning models, including DeepSeek-R1-Distill-Llama-8B, by supervised fine-tuning at known doses. This follows the Stage I/II protocol of Fragility, followed by one epoch of GRPO. Fragility reports average pass@1 inflation of 8.8% (Stage I) and 11.7% (Stage II), which is the signal the twin gap is built to catch.
The success criteria are three. The twin gap should keep high AUROC after GRPO while LiRA and Min-K% fall toward 50%. Its estimated inflation should track the injected dose monotonically. And the twin-transfer rate should be measured and should be small enough that the estimate stays well calibrated. If accuracy on twins rises nearly as much as accuracy on the leaked items, the estimator fails, and the proposal would have to fall back to purely temporal comparisons.
The second check is the decay fit on LiveBench release cohorts. It reports whether a drift-only model or a jump model at release dates explains the growing gap better on held-out months. The third check replays history and runs the calibrated simulator. Policies are compared on three measures: Kendall rank correlation, a measure of how closely two orderings agree, between the benchmark leaderboard and a leaderboard computed on items that were never released; discounted evaluations served; and time until that rank correlation falls below a preregistered ruin threshold. All comparisons are at equal generation budget.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Optimal dividends in the dual model under transaction costs
- ON OPTIMAL DIVIDENDS IN THE DUAL MODEL
- On the Bail-Out Optimal Dividend Problem
- Optimal dividends and capital injection: A general Lévy model with extensions to regime-switching models
- On the optimality of periodic barrier strategies for a spectrally positive Lévy process
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- A benchmark of expert-level academic questions to assess AI capabilities
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs