When a language model has seen benchmark questions during training, its score on that benchmark goes up for the wrong reason. Adaptive testing, which picks each next question based on how the model did so far, makes this worse in a quiet way. It deliberately asks questions near the model's estimated skill level, and a few memorized hard questions can push that estimate upward without anyone noticing. This proposal flips the question-picking rule. To estimate skill, the right questions are the ones a model of that skill gets right about half the time. To catch leakage, the right questions are the ones it should usually get wrong: questions calibrated as clearly above the model's ability. An honest model rarely solves them. A model that memorized them solves them almost every time. A correct answer there is surprising, and surprise is evidence.
We propose a black-box audit that needs only right-or-wrong answers. It first estimates a model's ability on anchor items that are unlikely to have leaked, drawn from recent or held-out pools. It then spends a fixed query budget on suspect benchmark items whose calibrated difficulty sits above that ability line, where leaked and honest response patterns differ most. The result is two numbers per model under a stated budget: a leakage-adjusted ability score and a calibrated test of whether contamination inflates the reported score past a tolerance. Because the signal is excess accuracy rather than token probabilities, we expect it to survive the reinforcement-learning post-training that has been shown to erase probability-based leakage signals.
Core Claim
The claim is narrow and testable. Under an item response model with a "leaked item" mixture component, the evidence a single question gives about leakage depends on where its difficulty sits relative to the model's ability, and it peaks well above the point that standard adaptive testing targets. Choosing questions on that basis, rather than sampling uniformly or reusing the skill-estimation rule, should reach a given detection power with substantially fewer queries.
Concretely, let $\theta$ be the model's ability, $b_j$, $a_j$, $c_j$ the calibrated difficulty, discrimination and guessing floor of item $j$, and $z_j \in \{0,1\}$ an unobserved indicator that item $j$ leaked into this model's training. We model correctness as
$$ P(y_j = 1 \mid \theta, z_j) = z_j\,(1-\epsilon) + (1 - z_j)\,\Big[c_j + (1-c_j)\,\sigma\big(a_j(\theta - b_j)\big)\Big], $$
where $\sigma$ is the logistic function, $\epsilon$ is a small learned rate at which memorized items still fail, and $z_j$ is Bernoulli with model-level leak fraction $\alpha$. The bracketed term is the standard three-parameter logistic model used by Adaptive Testing for LLM Evaluation (ATLAS). Items are scored by the expected divergence between the two branches,
$$ s_j(\theta) = \mathrm{KL}\big(\mathrm{Bern}(p_j(\theta)) \,\|\, \mathrm{Bern}(1-\epsilon)\big), $$
where $p_j(\theta)$ is the clean-branch probability. This score grows as $p_j$ falls toward the guessing floor. A worked case shows the size of the effect. On a four-choice item with $\epsilon = 0.01$, an item the model should answer with probability 0.625 yields about 1.1 nats of separation. That is roughly the point a Fisher-information rule picks. An item it should answer with probability 0.3 yields about 2.6 nats. Since required audit size scales inversely with per-query separability, as in the $\alpha\rho\sqrt{m}$ detectability law of When Is Benchmark Contamination Detectable?, this predicts roughly a two- to threefold reduction in queries for equal power. The bound is optimistic: it ignores error in $\theta$ and in the item parameters, and the experiments must show how much of it survives.
Relation to Prior Work
Two lines of work motivate the gap from opposite sides. ATLAS adapts computerized adaptive testing to language models and cuts evaluation cost by up to 90% by choosing items with maximal Fisher information at the running ability estimate. It treats item parameters as fixed across models and has no notion of an item being easy for one model because that model saw it. Our proposal keeps its calibration machinery but adds the leak branch and a second, separate selection rule. On the auditing side, When Is Benchmark Contamination Detectable? proves that detectability is governed by contamination fraction times separability times $\sqrt{m}$, and supplies a simulation planner for choosing audit size $m$. That planner assumes items are drawn passively. We attack the separability factor directly by choosing where to sample, and we reuse their planner to calibrate our tests.
The detection methods themselves constrain the design. ConStat detects contamination as a performance gap between original and rephrased items, after correcting for difficulty using about 20 reference models. It is black-box like ours, but it evaluates fixed item sets uniformly. PaCoST runs a paired test on stated confidence for original versus rephrased items. Detecting Data Contamination in LLMs via In-Context Learning (CoDeC) watches whether few-shot context lowers confidence on memorized items. RADAR classifies recall versus reasoning from internal activations. The last three need log-probabilities or internals.
The decisive constraint comes from On The Fragility of Benchmark Contamination Detection in Reasoning Models. It shows that one epoch of GRPO, a reinforcement-learning post-training method, collapses probability-based detectors toward chance. It also shows that contamination with chain-of-thought solutions transfers to rephrased siblings, which defeats every rephrasing-based check they tested. Yet in both settings accuracy on the benchmark stays inflated. That is why our signal is accuracy relative to an independent anchor, and not confidence or rephrasing.
The sampling ideas come from pure-exploration bandits, the setting where a learner spends a budget only to make a decision. The thresholding algorithm APT from An optimal algorithm for the Thresholding Bandit Problem spends queries on options whose mean is close to a threshold. LUCB from PAC Subset Selection in Stochastic Multi-armed Bandits samples the weakest current winner and the strongest current loser. Information Complexity in Bandit Subset Selection shows that the divergence between hypotheses at the decision boundary sets sample complexity. Our item score is that same principle applied within a single model: spend queries where the two hypotheses, leaked and honest, disagree most. The 3-SAT study shows that model accuracy collapses to about 10% at the hard phase transition, which tells us shortcut success concentrates away from the true difficulty boundary. We do not use 3-SAT itself, since synthetic formulas are not the contamination problem we target.
Research Program
The work has three parts.
First, we build anchored calibration. Item parameters for suspect pools (MMLU, ARC, GSM8K, HellaSwag) come from the released ATLAS item banks where available. Anchor items come from pools unlikely to be in pretraining for the audited models: LiveBench questions released after the model's cutoff, MMLU-CF, and WMDP, which PaCoST used as a temporally clean set. Anchors are placed on the same scale through shared reference models. Ability $\theta$ is estimated from anchors only, with ATLAS-style Fisher selection, so leaked suspect items cannot inflate it.
Second, the audit stage spends the remaining budget on suspect items chosen by $s_j(\hat\theta)$. The score is discounted by the posterior spread of $\hat\theta$, so the method does not over-trust a noisy ability estimate. We fit $\alpha$, $\epsilon$ and $\theta$ jointly by variational inference. This yields a leakage-adjusted score: expected suspect-pool accuracy with every $z_j$ set to zero. It also yields a test of $H_0: \alpha \le \alpha_0$, whose null distribution is calibrated with the two-stage simulation planner of Shihab et al.
Third, for leaderboards with many models, a second layer treats each model as an arm whose mean is its estimated inflation, the gap between reported and leakage-adjusted score. APT then allocates audit queries across models to decide which exceed a tolerance $\tau$. Models clearly clean or clearly inflated stop receiving queries early.
The main analysis compares the predicted per-query separation from the KL score with what we observe. It also measures how sensitive the method is to miscalibrated anchors, and it identifies the regime where the method should fail: chain-of-thought contamination broad enough to raise anchor performance too. In that regime we would argue the model has acquired genuine ability on that skill.
Baselines
The naive reference point is to report raw suspect-pool accuracy, or ATLAS ability estimated on the suspect pool, with no audit. It is run on the same models and item banks, and it defines how large the inflation is that we must remove. Beating it means recovering ability within the error of a never-contaminated twin model, where raw accuracy misses by the injected inflation. For reference, the Fragility paper reports 8.8% average pass@1 inflation from supervised contamination.
The strongest black-box detection competitor is ConStat, run on a uniform random subsample of the same query budget, with the same reference models for difficulty correction. ConStat reports detecting 98.4% of sample-specific contamination. That figure uses full benchmarks; no reported result was found on Lacuna for ConStat at budgets of a few hundred queries. Beating it means equal or higher detection AUROC at half the queries or fewer, or higher AUROC at matched budget, on the same injected-contamination models.
The strongest budgeting competitor is the power-calibrated uniform audit of Shihab et al., using their planner to choose $m$. They report 0.785 to 0.92 power at a 0.80 target, where plug-in Gaussian budgets failed in 9 of 9 cases. We give it the same calibration data. Beating it means hitting 0.80 power with a smaller $m$ across the same grid of contamination fractions.
Where log-probabilities are available, CoDeC and PaCoST serve as white-box upper references rather than competitors. CoDeC reports 99.9% dataset-level AUC on pretraining corpora. We expect them to win before reinforcement learning and lose after it.
Two ablations isolate the key ingredient. The first keeps our mixture model and anchors but selects suspect items by Fisher information, as ATLAS does. Any gain over it is due to above-the-line selection alone. The second keeps our selection but estimates $\theta$ from the suspect pool itself, which measures what the anchors contribute.
Why not just…?
Why not just run ATLAS and then flag items the model answers "too well" with a standard person-fit statistic? This is the most serious alternative, because person-fit checks exist exactly to catch test-takers who know answers they should not. The weakness is that an adaptive test chooses items the model solves about half the time, where a correct answer is barely surprising. The flagging then works on low-information data. The Fisher-selection ablation is this alternative in all but name. If it matches our method at equal budget, the selection rule adds nothing, and we would report that.
Why not just audit a larger uniform sample, sized by the power planner? The planner is the right tool and we use it. But the $\alpha\rho\sqrt{m}$ law means a uniform audit pays for low separability with quadratically more queries. Most uniformly drawn items sit below the model's ability, where leaked and honest models both answer correctly. The direct comparison at the 0.80 power target settles whether choosing items beats buying more of them.
Why not just retire old benchmarks and use rolling ones like LiveBench? For new evaluations this is often right, and our method depends on such pools for anchors. But LiveBench covers only programmatically checkable tasks and needs about a sixth of its items replaced every month. Scores already published on MMLU, ARC or GSM8K still need checking, and CoDeC finds that leakage signals appear after a model has seen only about 2% of training data, so fresh items do not stay fresh for long. Our audit converts a small anchor pool into a check on a large legacy pool.
Why not just compare original and rephrased questions? This fails exactly where audits matter most. The Fragility paper shows that chain-of-thought contamination transfers to rephrased siblings and pushes all ten detectors it tested to 50 to 58% AUROC. Our anchors come from different sources and dates, so the inflation must appear on anchors too before it can hide. Experiment three tests this directly.
Proposed Experiments
The first experiment injects contamination under control. We fine-tune open base models on known random subsets of MMLU, ARC, GSM8K and HellaSwag at leak fractions of 2, 5, 10 and 20%, with a clean twin trained on matched non-benchmark text. We report detection AUROC versus query budget, the queries needed for 80% power, and the absolute error between leakage-adjusted ability and the clean twin's ability. For context, ATLAS reports 0.157 mean absolute ability error with 41 items on clean HellaSwag. The decisive check is whether above-the-line selection beats both uniform ConStat and the Fisher-selection ablation at budgets of a few hundred queries. It should do so by roughly the factor the KL calculation predicts.
The second experiment repeats this pipeline after one epoch of GRPO, following the Stage I protocol of the Fragility paper on AIME, AMC, GPQA Diamond, OlympiadBench and Minerva Math. That paper reports LiRA AUROC falling from 89.1% by about 14 points. Our prediction is that white-box detectors collapse while our accuracy-based audit loses little. Failure here would mean reinforcement learning also erodes the excess accuracy, which would be informative in itself.
The third experiment reproduces the Stage II chain-of-thought contamination setting, which the Fragility paper ran on DeepSeek-R1-Distill-Llama-8B and found an 11.7% benchmark boost with detectors at 50 to 58%. We test whether anchor-relative excess accuracy recovers detection. If anchors rise along with suspect items, we report that boundary honestly.
The fourth experiment audits publicly released open models with no injected leakage and runs the APT leaderboard layer. We check whether the models we flag agree with prior findings, such as ConStat's reported gaps of about 6.6% for Llama-3-70b on ARC and 8.3% for Mistral-7b on GSM8K. We also measure the fraction of total queries APT saves relative to uniform per-model audits that reach the same classification accuracy against $\tau$.
References
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
- ConStat: Performance-Based Contamination Detection in Large Language Models
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Detecting Data Contamination in LLMs via In-Context Learning
- RADAR: Mechanistic Pathways for Detecting Data Contamination in LLM Evaluation
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- An optimal algorithm for the Thresholding Bandit Problem
- PAC Subset Selection in Stochastic Multi-armed Bandits
- Information Complexity in Bandit Subset Selection
- Can Large Language Models Reason? A Characterization via 3-SAT
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark