Contaminated Answers Arrive Too Fast: Detecting Benchmark Leakage From How Little Reasoning a Model Needs

A model that has seen a test question during training does not have to work it out; it can recall the answer. Computer science has a precise name for this situation, the space-time tradeoff: you can answer a query quickly if you have paid in advance to store the answer, or you can store little and compute at query time. We propose to use this principle as a contamination detector for reasoning models. For each benchmark item, we measure how much reasoning, counted in generated tokens, the model needs before it answers correctly. We then compare that amount with what the item's difficulty predicts. A model that solves a hard multi-step problem after almost no reasoning is behaving like a lookup table, not a solver. That anomaly is our contamination signal.

This matters because the cheap detectors currently in use are breaking. Detectors that flag items where the model is unusually confident lose most of their power after a single epoch of reinforcement learning. When a model is trained on leaked worked solutions instead of leaked answers, ten standard detectors fall to near chance (On The Fragility of Benchmark Contamination Detection in Reasoning Models). Our bet is that reinforcement learning can flatten a model's token probabilities but cannot easily hide the fact that a stored answer is cheap to produce. The detector works per item and needs only the ability to generate text, so it can audit models whose training data is closed.

Core Claim

The claim is that difficulty-adjusted minimal reasoning budget is a contamination signature that survives reinforcement-learning post-training better than probability-based detectors. Here "minimal reasoning budget" means the smallest number of thinking tokens after which the model, forced to commit, answers correctly. We define it for item $i$ as

$$ b^*_i = \min \{ b \in \mathcal{B} : \hat{p}_i(b) \ge \tau \}, $$

where $\mathcal{B}$ is a grid of reasoning-token budgets (for example 0, 16, 64, 256, 1024, unconstrained) and $\hat{p}_i(b)$ is the fraction of $k$ sampled runs that are correct when reasoning is cut off at $b$ tokens and the model is forced to state an answer. The threshold $\tau$ is a pass-rate cutoff such as 0.5. The contamination score is

$$ s_i = \log \hat{b}_i - \log b^*_i, $$

where $\hat{b}_i$ is the budget the item is expected to need. On procedural tasks, $\hat{b}_i$ is predicted from the known number of reasoning steps. On natural benchmarks, it is predicted from the budgets that a fleet of reference models needs on the same item. Large positive $s_i$ means the model answers much faster than the item's difficulty warrants.

The analogy to indexing does real work here. Analyses of learned indexes show that constant-time lookup is possible only if the structure stores close to one piece of information per key, while compact structures must pay time that grows with the data (Lower Bounds for the Algorithmic Complexity of Learned Indexes; Beyond Logarithmic Bounds: Querying in Constant Expected Time with Learned Indexes). A memorized benchmark item is the extreme case: the weights store the answer, so the time to retrieve it should be roughly constant regardless of how many reasoning steps the problem logically requires. Genuine solving should cost time that grows with step count. We use the analogy only to predict a shape: flat budget curves for leaked items and rising curves for clean ones. We do not claim a formal lower bound for transformers.

Relation to Prior Work

Two papers motivate the proposal. The fragility study fine-tunes reasoning models on leaked benchmark items and finds three things: contamination inflates pass@1 by 8.8% on average; the LiRA detector, which scores items by how much more likely the model makes them than reference models do, starts at 89.1% AUROC and drops about 14 points after one epoch of GRPO; and contamination with chain-of-thought traces drives ten detectors to 50–58% AUROC (On The Fragility of Benchmark Contamination Detection in Reasoning Models). All of those detectors read token probabilities. Ours reads the cost of successful behavior instead. The second motivating paper finds that Qwen2.5-Math-7B reaches nearly 40% accuracy on MATH-500 given only the first 40% of each problem, while Llama3.1-8B scores near zero (Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination). That is the input-side version of our idea: correct answers from too little question. We probe the output side, correct answers from too little reasoning, and we normalize by item difficulty.

Three papers constrain the design. ConStat compares a model's benchmark score with its score on rephrased reference sets, corrected for hardness using reference models (ConStat: Performance-Based Contamination Detection in Large Language Models). We borrow its reference-fleet hardness correction, but ConStat produces a model-level verdict, while we need item-level scores and add the compute axis. Adaptive testing with item response theory, a statistical model that assigns each question a difficulty, flags models that solve hard items while missing easy ones (Efficiently Measuring the Cognitive Ability of LLMs: An Adaptive Testing Perspective). That signal uses correctness alone, so a genuinely strong model that solves hard items looks the same as a contaminated one; compute is meant to separate them. Finally, Inverse Scaling in Test-Time Compute shows that accuracy is not monotone in reasoning length. We therefore define $b^*_i$ as the first budget that crosses the threshold, not as the point of peak accuracy, and we analyze non-monotone curves separately.

Two further papers matter for the evaluation. Test of Time: Rethinking Temporal Signal of Benchmark Contamination finds that paraphrased leakage erases the signal of accuracy dropping on post-cutoff data, and that influence-function tracing back to the source document falls from 77.5% to 17.5% under transformation. This makes paraphrased contamination the hardest case we must test. A Narrowing Geometry in Contaminated Reasoning finds that leaked items collapse internal activations into a few dominant directions. That is a mechanistic counterpart to our behavioral shortcut, and it gives us a white-box check. It also supplies MSCalc, a synthetic multi-step arithmetic task with controlled chain-of-thought length.

Research Program

The work has three parts.

First, we build ground-truth contamination. Starting from open reasoning models used in the cited studies, such as DeepSeek-R1-Distill-Llama-8B and Qwen2.5-Math-7B, we train on a random half of each benchmark's items under four regimes. The regimes are verbatim question-answer fine-tuning; fine-tuning followed by one or more epochs of GRPO on clean data, the cloaking setting from the fragility study; fine-tuning on full chain-of-thought solutions; and fine-tuning on LLM paraphrases of the items. The held-out half serves as clean negatives, so every item has a known label.

Second, we compute budget curves. For each item and budget we sample $k$ runs (for example $k=8$), truncate reasoning at the budget, append a forced-answer cue, and grade the result. We estimate $\hat{b}_i$ in two ways. On MSCalc and DyVal, a generator of reasoning problems from random graphs with controllable depth (DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks), we regress budget on the known step count using uncontaminated runs. On GSM8K, MATH-500, AIME, AMC and GPQA, we fit $\hat{b}_i$ from a fleet of reference models that never saw the leaked split.

Third, we test whether the signal survives in the wild. We apply the detector to off-the-shelf models where prior evidence points to leakage, comparing Qwen2.5-Math-7B and Llama3.1-8B on MATH-500, and we use RandomCalculation as a leakage-free control on which no model should score high. In the controlled models we also measure the activation-concentration index from the narrowing-geometry paper. This tells us whether a high $s_i$ coincides with the low-rank collapse that paper describes, which would be evidence that the behavioral and mechanistic shortcuts are the same phenomenon.

Baselines

The naive reference point assigns contamination scores using only item difficulty, or at random. It sets the floor at 50% AUROC and checks that our difficulty estimates do not leak labels on their own.

The strongest existing item-level competitor is the set of probability-based membership detectors evaluated in the fragility study, chiefly LiRA together with loss and Min-K%. The cited numbers are LiRA at 89.1% AUROC after fine-tuning, falling about 14 points after one GRPO epoch, and all ten detectors at 50–58% AUROC under chain-of-thought contamination (On The Fragility of Benchmark Contamination Detection in Reasoning Models). We will rerun these detectors ourselves on our contaminated checkpoints, with the same items and reference models, rather than relying on the reported figures. Beating them means matching LiRA within a few points under plain fine-tuning and clearly exceeding it after GRPO and under chain-of-thought contamination, for example by staying above 70% AUROC where LiRA falls toward 60%. We fix that threshold before running.

The second strong competitor is the prefix-truncation test from Reasoning or Memorization?, which feeds the first 40% of the question. Converted to an item-level score, it is accuracy on truncated prompts, normalized by the reference fleet. It is cheap, behavioral, and close in spirit to our method, so it is the most dangerous baseline. We give it the same sampling budget, measured in total generated tokens, as our method.

The third competitor is a correctness-only difficulty-anomaly score in the style of ConStat and item-response-theory person-fit, fitted on the same reference fleet. ConStat reports detecting 89% of syntax-contaminated and 98% of sample-contaminated models, but these are model-level detection rates. We found no reported item-level result for it on Lacuna. This baseline doubles as the key ablation: it uses the same difficulty normalization without the budget axis.

Two more ablations isolate the remaining ingredients. A raw $b^*_i$ score without difficulty normalization tests whether normalization is necessary. A combined score that adds our budget signal to the prefix test tests whether the two signals are complementary.

Why not just…?

Why not just use the prefix-truncation test? It is the real threat. If an item's answer can be recalled from a fragment of the question, the item probably also needs little reasoning, so the two signals may be largely redundant on verbatim contamination. They should diverge under paraphrased and chain-of-thought contamination, where the model may never have seen the exact opening of the question but did learn a shortcut solution. If our score does not beat the prefix test in those two regimes at equal token budget, we will report the budget curve as a complement to the prefix test rather than a replacement.

Why not just use likelihood-based detectors like Min-K% or LiRA? The fragility study already shows that they lose much of their signal after a single GRPO epoch and sit near chance under chain-of-thought contamination. Reinforcement learning compresses exactly the probability gap these detectors read. Our GRPO-cloaking regime tests directly whether behavioral cost resists the same compression.

Why not just fit item response theory or ConStat and flag hard items the model solves? Solving a hard item is also what a genuinely capable model does, and these methods cannot tell the two apart from correctness alone. The correctness-only ablation measures how much the compute axis adds. If it adds nothing, the idea fails.

Why not just switch to procedural benchmarks? DyVal and RandomCalculation cover formal domains, but not open-ended benchmarks like GPQA or Humanity's Last Exam that people still report. We use procedural tasks as the clean control and the step-count calibration, not as the target.

Decisive Checks

Four results would settle the question.

The first is the slope test. On MSCalc and DyVal, $b^*$ should rise with step count for clean items and stay nearly flat for leaked items, and this separation should persist after GRPO. If the leaked curves also rise with step count, the core mechanism is wrong.

The second is item-level AUROC after GRPO cloaking and under chain-of-thought contamination on GSM8K, MATH-500 and AIME/AMC. This is the comparison against rerun LiRA and the prefix test.

The third is paraphrase contamination, the regime where Test of Time shows other temporal and tracing signals vanish. We expect our detector to degrade there too and will report by how much.

The fourth is the field check. Scores should be high for Qwen2.5-Math-7B on MATH-500 relative to Llama3.1-8B, and uniformly low on RandomCalculation for all models.

We also report cost in generated tokens per audited item. The intended selling point is an audit that needs only generation access and costs a few thousand tokens per item, compared with influence-function tracing that could cover only 40 papers.