When a model has trained on a benchmark question together with its answer, the cleanest evidence of that is not how confident it looks on the question. It is what the model does when the question is changed just enough that the old answer becomes wrong. A model that learned to solve the problem should follow the change and produce the new answer. A model that memorized the item will often give back the answer it stored. We propose to audit contamination by building, for each benchmark item, a small set of answer-flipping near misses: minimal edits to the question, verified by a program to change the correct answer, and chosen to stay as close to the original as possible. The audit signal is answer persistence, meaning how often the model returns the original answer to an edited question, compared against the rate expected from models known to be clean. This signal needs only the generated text, not token probabilities.
The timing matters because the standard detectors have stopped working on reasoning models. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows two failures. A short phase of clean reinforcement learning after contaminated fine-tuning collapses the log-probability gap that detectors rely on. Contamination with chain-of-thought solutions (training on worked, step-by-step solutions) teaches the model the procedure well enough that it is equally confident on unseen similar items. In the second case, ten detectors averaged 50–58% AUROC. Both failures attack confidence signals on unchanged text. Our bet is that memorized answers stay recoverable even after confidence has been flattened, and that an edit which keeps the question's surface but changes its answer is the stimulus most likely to bring them out.
Core Claim
The claim has three parts. First, answer persistence on verified answer-flipping near misses separates contaminated items from clean ones even after reinforcement learning has erased likelihood-based signals. Second, the separation depends on how near the edit is. Edits far from the original look like new problems, and a clean model and a contaminated model handle them alike. The nearest edits trigger recall of the stored answer. Third, the nearness has to be paired with verification. Near-miss edits that do not actually change the answer would score a correct solver as "persistent," so unverified nearest-neighbour selection should damage the detector.
The third part is the benchmark-auditing version of the false-negative dilemma that Sampling Better Negatives for Distantly Supervised Named Entity Recognition identified in entity recognition. Its Top-Neg method picks the negatives most similar to the positives because those carry the most training signal, and it notes that the most similar candidates are also the ones most likely to be mislabeled. We take the same structure into evaluation. The most informative probe is the closest one that is still truly different, and the programmatic answer check plays the role that a clean label set plays in their setting.
Relation to Prior Work
The contamination detectors closest to this proposal all compare an original item against a perturbed version. PaCoST paraphrases each question and runs a paired test on the model's stated confidence that the answer is correct. ConStat rephrases or synthesizes reference benchmarks and fits a difficulty curve across clean models to measure a performance gap. Both are the motivation for this proposal, because they show that perturbation pairs carry contamination signal. They also constrain it. Their perturbations keep the answer fixed (PaCoST) or are sampled without regard to how close they sit to the original (ConStat). The fragility paper explains why answer-preserving paraphrase fails on chain-of-thought-trained models: the model solves the paraphrase as easily as the original. We keep ConStat's idea of calibrating against clean reference models, but replace accuracy on loosely related items with a sharper outcome. Did the model output the specific old answer on an item where that answer is provably wrong?
The likelihood detectors, Min-K% Prob and Min-K%++, score whether a text was seen by looking at its lowest-probability tokens. Min-K%++ argues that training makes seen samples local maxima of the model's probability landscape. "Local maximum" is a claim about the model's behaviour in a small neighbourhood of a seen item. Our probe tests that neighbourhood directly in output space instead of through the logits that reinforcement learning compresses.
Dynamic benchmarks such as DyVal and INVITE build fresh items. DyVal does it from graphs with added distractors. INVITE swaps an entity for another of the same category to create false-premise questions. Both avoid contamination by being new. Neither picks its perturbations by nearness to a specific leaked item, and neither uses perturbations to audit whether that item leaked. MASS-DPO supplies a secondary method: it selects a small, non-redundant set of informative negatives. We borrow its point that a few well-chosen negatives can replace many, but only as a budget-reduction step. The core selection rule is nearness plus verification.
Research Program
The first component is a probe generator. We will start with numeric and condition-based math problems, where the correct answer to an edit can be recomputed exactly. For each item, an LLM rewrites the problem into a parameterized template, with numbers and conditions as variables. A symbolic solver checks that the template reproduces the original answer, and items whose templates fail this check are discarded. We then enumerate candidate edits, recompute their answers, and keep only edits whose answer differs from the original.
Candidates are ranked by nearness to the original in three ways: token edit distance, embedding distance under a fixed public encoder, and the change in difficulty measured on clean reference models. Using a fixed public encoder keeps the method usable against black-box systems. For each item we keep a few of the nearest candidates that are verified and difficulty-matched. As an optional step, we select a small non-redundant subset in the style of MASS-DPO to reduce query cost.
The second component is the statistic. Let $\pi_i$ be the fraction of sampled responses to item $i$'s probes that reproduce the original answer. Let $\bar\pi_i$ be the same quantity averaged over a pool of clean reference models. The item score is $\pi_i - \bar\pi_i$. Averaged over a benchmark, it gives both a membership score for AUROC and a paired test for flagging a whole benchmark as contaminated.
The third component is a sweep over nearness, which tests the claim that separation depends on how close the edit is. We will bin probes by distance and plot the contaminated-minus-clean persistence gap per bin. The predicted shape is a gap that rises toward the nearest bin. If clean models also start missing very subtle edits, the gap may flatten at the closest distances. The clean-model baseline $\bar\pi_i$ is there to absorb that effect.
Baselines
The naive reference point is chance AUROC of 0.5, together with accuracy-only scoring on the same edited probes, which simply checks whether the model gets the edit wrong. This shows whether persistence, meaning a wrong answer that equals the old answer, adds anything beyond ordinary failure.
The strongest existing white-box detectors are LiRA, Loss and Min-K%, run exactly as in On The Fragility of Benchmark Contamination Detection in Reasoning Models, with Min-K%++ added. We use the same contaminated checkpoints and the same member/non-member splits. That paper reports LiRA at 89.1% AUROC after contaminated fine-tuning, a drop of about 14 points after one epoch of GRPO, and an average of 50–58% AUROC across ten detectors under chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. Beating them means clearly exceeding the post-GRPO LiRA figure and the 58% ceiling under chain-of-thought contamination, with confidence intervals from item-level bootstrap.
The strongest perturbation-based competitors are PaCoST and ConStat. We give them the same items and the same per-item query budget, and give ConStat the same clean reference pool. PaCoST receives paraphrases from the same generator LLM. ConStat is reported in its original setting as finding an 8.3% gap for Mistral-7b-v0.1 on GSM8K. We have found no reported AUROC for PaCoST or ConStat on the fragility paper's reasoning-model checkpoints on Lacuna, so we will produce those numbers ourselves.
The key ablations isolate each ingredient. Random verified answer-flipping edits (no nearness ranking) are the ConStat-like control and test whether selection matters. Nearest edits without verification test the false-negative effect. Answer-preserving nearest edits isolate the flip itself.
Why not just…?
Why not just perturb numbers at random and measure the accuracy drop, as ConStat effectively does? This is the most serious threat, and it may come close to our method. The random-verified-edit ablation is built to settle it. If persistence on random flips matches persistence on nearest flips, the selection idea fails, and the contribution shrinks to the persistence statistic itself. The fragility paper's observation that chain-of-thought contamination lets models generalize to distributionally similar items suggests that distant edits will be solved by contaminated and clean models alike. That is a prediction we can falsify, not a result we can already claim.
Why not just use Min-K%++ or LiRA? The fragility paper reports that these fall to near-chance AUROC after GRPO and under chain-of-thought contamination. They also need log-probabilities, which commercial APIs often withhold. We will still run them on identical checkpoints, so any advantage we claim is measured head-to-head.
Why not just evaluate on fresh post-cutoff data like LiveBench? Fresh data gives a clean score, but it cannot tell you which existing reported results were inflated. LiveBench also notes the ongoing manual effort of refreshing its questions. Our goal is to audit benchmark claims that have already been made, which complements fresh evaluation rather than competing with it.
Proposed Experiments
The controlled study reproduces both stages of the fragility paper on AIME, AMC, OlympiadBench and Minerva Math, restricted to items whose templates the solver can verify. The first stage is contaminated fine-tuning followed by clean GRPO. The second is chain-of-thought contamination of DeepSeek-R1-Distill-Llama-8B. The primary metric is item-level membership AUROC, with benchmark-level flagging power at a fixed false-positive rate as a secondary metric. We will report the share of items for which verified templates could be built, because that coverage limits where the method applies. GPQA Diamond is left out of the main study because its answers cannot be recomputed this way. We will try LLM-proposed condition edits on it only as an exploratory, separately reported arm.
A natural-contamination check applies the probe to GSM8K. It tests whether persistence is elevated for Mistral-7b-v0.1, which ConStat flagged with a gap of about 8.3%, relative to clean reference models.
The decisive result has three conditions. Nearest verified flips must beat both random verified flips and the post-GRPO LiRA figure. Unverified nearest edits must lose much of that advantage. The persistence gap must widen as nearness increases. If persistence works but nearness does not matter, we will report the simpler method honestly.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Sampling Better Negatives for Distantly Supervised Named Entity Recognition
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Detecting Pretraining Data from Large Language Models
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark