Stale Consensus: Catching Benchmark Contamination When Many Samples Agree on the Wrong Problem

Many methods that check facts by counting agreement rest on one bet: if a claim keeps showing up across independent sources or independent samples from a model, it is probably true, because errors are random and rarely repeat. Benchmark contamination breaks that bet. Contamination means the test questions, or their solutions, leaked into a model's training data. A model that has memorized a test problem will repeat the memorized answer in sample after sample, and a majority vote will read that repetition as confidence. We propose to turn this failure into a detector. Take a benchmark problem and change it so its correct answer must change, for example by altering a number in a math word problem and recomputing the answer. Then sample the model many times on the edited problem and count how often its answers and intermediate values still match the original problem. We call this signal stale consensus. A model that reasons will follow the edit. A model that recalls a memorized solution will keep converging on content that no longer belongs to the question.

The signal matters because the usual contamination detectors are failing on reasoning models. The main existing tools read token probabilities, and On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that reinforcement-learning post-training pushes them to near chance. Stale consensus needs only sampled text, it gives a score for each item rather than for a whole dataset, and it controls for problem difficulty by design, because the edited problem is as hard as the original. The same measurements also test the independence assumption behind frequency-based validation, for which the research file contains no direct evidence: how often does self-consistency confidently choose a memorized answer that is wrong for the question being asked?

Core Claim

Our claim is that on reasoning benchmarks, contamination leaves a characteristic trace in the frequencies of repeated samples. It is not low diversity across samples, and it is not a drop in accuracy under rewording. It is persistence of the original item's specific content under an edit that changes the answer. Concretely, let $x$ be a benchmark item and $x'$ an edited version whose verified answer differs. Let $V(x)$ be the set of numeric values that appear in a reference solution of $x$ but not in a reference solution of $x'$; this includes the original final answer and its intermediate results. Draw $N$ completions $y_1,\dots,y_N$ on $x'$. The stale-consensus score is

$$S(x) = \frac{1}{N}\sum_{i=1}^{N}\Big(\lambda\,\mathbb{1}[\mathrm{ans}(y_i)=\mathrm{ans}(x)] + (1-\lambda)\,\frac{|\mathrm{vals}(y_i)\cap V(x)|}{|V(x)|}\Big),$$

where $\mathrm{ans}(\cdot)$ extracts a final answer, $\mathrm{vals}(\cdot)$ extracts the numeric values a completion asserts, and $\lambda \in [0,1]$ weights final-answer reuse against intermediate-value reuse. The score treats each extracted value as an atomic claim and counts how many samples repeat it. This is the redundancy counting of A Probabilistic Model of Redundancy in Information Extraction and Atomic Self-Consistency for Better Long Form Generations, but here repetition counts as evidence of memorization, not of truth.

Relation to Prior Work

Three papers motivate the proposal by showing where current detectors break. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates Qwen2.5-7B-Instruct through fine-tuning and then applies clean reinforcement learning with GRPO. One epoch of GRPO drops LiRA from 89.1% AUROC, and reference-free methods fall toward 50%. When the authors contaminate DeepSeek-R1-Distill models on reasoning traces, all ten detectors they test score 50–58% AUROC, because the model generalizes to similar unseen questions. Reasoning or Memorization? finds that Qwen2.5 reconstructs MATH-500 answers from 80% prefixes at 65.8% exact match, and that gains from spurious rewards disappear on the leakage-free RandomCalculation generator. Min-K% (Detecting Pretraining Data from Large Language Models) and Min-K%++ (Min-K%++) read token log-probabilities from a single forward pass. They are exactly the white-box tools that reinforcement learning hides, and we keep them only as reference points.

The closest existing work is Detecting RLVR Training Data via Structural Convergence of Reasoning. Its Min-$k$NN Distance samples 32 completions per prompt, measures edit distance between each completion and its nearest neighbor, and treats low diversity as a sign the prompt was seen in training. It averages 0.70 AUC and survives paraphrasing. Our method differs in what it measures and on which input. Min-$k$NN measures how much samples on the original prompt agree with each other. Easy problems also produce similar solutions, so that signal mixes contamination with difficulty. We sample on an edited prompt, and we measure agreement with content that the edit should have made obsolete, so an easy clean problem scores near zero.

CAP perturbs items without changing their meaning (shuffled options, altered irrelevant details, paraphrases) and compares aggregate accuracy consistency between training and test splits. It assumes memorization is brittle. Min-$k$NN's robustness to paraphrase and the generalization reported by the Fragility paper point the other way. Our design accepts both findings. A memorizing model may keep its accuracy under rewording yet still be anchored to the original content under an answer-changing edit. We therefore use edits that change the meaning, score single items rather than datasets, and inspect the content of the answers rather than their correctness.

Two recognition- and perturbation-based methods constrain the design. Data Contamination Quiz asks the model to pick the original item out of word-level perturbations while correcting for position bias. Data Contamination Calibration for Black-box LLMs uses random word swaps and recovers log-probabilities through the logit_bias API, which many endpoints do not offer. Both show that perturbation exposes memorization, but both work on surface wording rather than on reasoning content. Finally, Improving LLM Generations via Fine-Grained Self-Endorsement and Atomic Consistency Preference Optimization supply the method of splitting outputs into atomic facts and counting agreement among them. The latter explicitly warns that agreement signals truth only when the model is calibrated, and our second analysis measures directly how that assumption fails under contamination.

Research Program

We will build three components.

The first is an edit generator for math benchmarks. It rewrites numeric parameters in GSM8K and MATH problems and recomputes the answer, either with a symbolic program or by solving with a strong model and checking the result with a computer algebra system. The generator keeps only edits whose new answer differs from the original and whose reference solutions differ in at least some intermediate values. Problems whose structure cannot be edited this way, which will include many geometry and proof-style MATH items, are dropped and reported. For each kept item we produce two edits so the score can be averaged across them.

The second is an extractor that pulls asserted numeric values and final answers from reasoning traces. It reuses the atomic-claim approach of the self-endorsement work, restricted to numbers so that matching is exact and cheap.

The third is an amplification analysis. On edited items, we run majority voting and atomic self-consistency and record how often the consensus answer is the stale original answer. We then break this rate down by the item's contamination status.

Baselines

The naive reference points are random scoring and a within-prompt agreement score. The agreement score is the fraction of $N$ samples on the original item that share the majority final answer, which is essentially free once samples exist. If stale consensus cannot beat this score, the edit adds nothing.

The strongest black-box competitor is Min-$k$NN Distance from Detecting RLVR Training Data via Structural Convergence of Reasoning. That paper reports an average AUC of 0.70 across models from 1.5B to 32B parameters, 0.71 under paraphrasing, and 0.76 on OpenR1-Distill-7B. We will run it with the same $N$ and the same sampling temperature as our method. Because our method also spends samples generating on the edits, we will additionally give Min-$k$NN twice as many samples as our method uses per edit, so that neither method wins on budget. Beating it means higher item-level AUROC on the same contaminated and clean splits, with the gap holding on the RLVR-trained checkpoints that setting was designed for.

For white-box reference points we run Loss, Min-K%, Min-K%++, and LiRA where weights are available. The Fragility paper's reported numbers anchor these: 89.1% AUROC for LiRA before GRPO, and 50–58% AUROC for all ten detectors in its Stage II setting. For Stage II, beating the baselines means clearly exceeding 58% AUROC. We would regard 0.70 or higher as a meaningful result, although no reported result for a black-box method in that setting exists in the file.

We also run CAP's accuracy-consistency comparison adapted to single items: the drop in accuracy from $x$ to $x'$. This baseline isolates our key ingredient, because it uses the same edits and the same samples and discards only the stale-content count. Two further ablations use the same samples. The first sets $\lambda=1$, keeping only final-answer reuse, to test whether intermediate values add signal. The second replaces the answer-changing edits with CAP-style edits that preserve meaning, to test whether changing the answer is what matters. Where the endpoint allows it, we run Data Contamination Quiz on the same items as the strongest recognition-based black-box competitor.

Why not just…?

Why not just use Min-$k$NN, or answer agreement on the original prompt? Both methods measure how much samples agree, and agreement is driven by difficulty as well as by contamination. Level-1 MATH problems and short GSM8K problems yield near-identical clean solutions. We will test this directly by binning items by solve rate on a clean model and reporting each detector's AUROC within each bin. We predict that the agreement-based scores lose most of their signal within bins while stale consensus keeps it. If the within-bin gap does not appear, Min-$k$NN is the better choice because it is simpler.

Why not just measure the accuracy drop on edited variants, as CAP does? This is the real threat, because it uses the same edits. An accuracy drop is ambiguous, however. Edits can make a problem harder, and the Fragility paper finds that contaminated reasoning models generalize to similar questions, which may keep accuracy on edits high. Stale consensus instead asks where the wrong answers come from. The ablation on identical samples settles the question: if the per-item accuracy drop matches stale consensus in AUROC, the stale-content count is unnecessary, and we would report that result.

Why not just evaluate on fresh benchmarks such as RandomCalculation? Fresh sets avoid contamination but do not audit the existing results on GSM8K and MATH, and they cannot say which items leaked. Reasoning or Memorization? reports no per-item attribution on its clean benchmark. We use RandomCalculation for a different purpose: as a guaranteed-clean control set on which stale consensus should stay near zero.

Why not just use Data Contamination Quiz? That method detects recognition of wording through synonym swaps. After reinforcement learning on reasoning traces, what persists may be the solution rather than the question's wording. We run it as a baseline, and if it matches stale consensus at lower cost, we will say so.

Proposed Experiments

In the controlled setting, we reproduce the Fragility paper's two pipelines on AIME, AMC, Minerva Math, and a subset of MATH. The first pipeline contaminates Qwen2.5-7B-Instruct through fine-tuning and follows it with 0, 1, and 3 epochs of clean GRPO. The second contaminates DeepSeek-R1-Distill-Llama-8B on reasoning traces. We add the edit-compatible GSM8K items, with half the items in each benchmark contaminated. The primary metric is item-level AUROC. We also report true-positive rate at 5% false-positive rate, and compute cost per item measured against Min-$k$NN's reported 6.65 seconds.

In the natural setting, we score Qwen2.5 models on MATH-500, which is contaminated according to prefix reconstruction, against LiveMathBench and RandomCalculation, and we compare them with Llama models of matched accuracy.

The decisive checks are three. First, stale consensus must beat the per-item accuracy drop on identical samples; otherwise the stale-content count is unnecessary. Second, it must beat Min-$k$NN within difficulty bins. Third, it must stay clearly above 0.58 AUROC after GRPO, where the file reports that likelihood-based detectors fail. The amplification analysis then reports, for contaminated versus clean items, the fraction of edited problems on which majority voting and atomic self-consistency select the stale answer. This gives the first direct measurement of how often frequency-based validation certifies a memorized error.