When a math benchmark leaks into a model's training data, its scores stop measuring reasoning. The standard way to catch this checks whether the model assigns suspiciously high probability to the test text. A short round of reinforcement learning on clean data defeats that check almost completely. We propose a different test that never looks at probabilities. For each test problem, we write two "twins": versions with new numbers or surface details but the same solution structure. We then sample several solutions from the model for the original and for each twin. A model that reasons treats the original as just another member of the family. A model that memorized the original solves it in a collapsed, stereotyped way that its twins do not share. The statistic asks whether the original is a distributional outlier among its own twins.
Two things make this worth doing now. Leaderboards for reasoning models are built on reinforcement learning from verifiable rewards, and contamination detection has a documented blind spot after exactly that stage. Meanwhile, much of the evidence that small curated datasets beat large ones is measured on the same leak-prone benchmarks. An audit that survives reinforcement learning would also let us ask how much of a "quality over quantity" gain is real.
Core Claim
Reinforcement learning hides contamination by erasing the gap in log-probability between seen and unseen problems. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows this directly. It contaminates Qwen2.5-7B-Instruct during supervised fine-tuning, then runs one epoch of clean GRPO, a reinforcement learning method that scores groups of sampled answers against each other. Afterward, LiRA (a likelihood-ratio membership attack using reference models) loses about 14 AUROC points from its post-SFT 89.1%, and reference-free detectors fall to chance. The paper proves the cause: importance-ratio clipping shrinks the spread of loss on unseen problems.
Our claim is that this mechanism acts on likelihoods of fixed text, not on the content of sampled solutions. It should therefore leave intact a signal defined only by how sampled reasoning traces change when the problem is minimally edited.
We make "outlier among twins" precise with a two-sample distance. Let $T_x$ be $k$ traces sampled for problem $x$, embedded by a fixed text encoder. Let $x'$ and $x''$ be two verified twins, and let $D$ be the energy distance, a distance between two point clouds that works with small samples. The per-item score is
$$s(x) = D(T_x, T_{x'}) - D(T_{x'}, T_{x''}).$$
The second term is a built-in null. It measures how far apart two equally valid, never-seen versions of the same problem already sit. As a result, no clean reference model is needed, and differences due to changed numbers cancel out. A clean model should give $s(x)\approx 0$. A model that replays a memorized solution should give $s(x) > 0$.
The score deliberately measures the part of contamination that matters for validity: the part that does not transfer to twins. Suppose contamination taught a schema that also solves the twins. The Stage II results of the Fragility paper hint this happens with chain-of-thought leakage. In that case the twins inherit the inflated accuracy, and the "contamination" has become generalization. The score should then read near zero.
Research Program
Twin generator. We restrict to numeric-answer problems from MATH-500, AMC, AIME and OlympiadBench. An LLM rewrites each problem with new constants and surface entities. The rewrite must keep a program-checkable answer, and we accept it only if a recomputed solution verifies. Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination motivates this step. It found that LLM-transformed questions remove the post-cutoff decay seen in verbatim questions, and that its influence-function trace back to source documents falls from 77.5% to 17.5%. This suggests edits break retrieval. We use that only as a method precedent and verify twin quality ourselves. The validation model is Llama3.1-8B. Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination finds this model near zero under partial-prompt completion on MATH-500. It should show equal accuracy on originals and twins, and $s(x)$ centered at zero.
Brittleness as the design principle. Twins exploit the reasoning limitations this literature documents. Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones? shows models fail simple variants of problems they solve. BeyondBench shows models above 90% on GSM8K roughly halving on uncontaminated algorithmic variants. Both papers report accuracy gaps aggregated over a benchmark. We turn the same sensitivity into a per-item distributional test.
Distributional machinery. The distance estimators come from the distributional-metrics literature. Hacking Generative Perplexity shows energy distance and MAUVE catch degenerate output distributions that scalar perplexity rewards. Exploring Precision and Recall to assess the quality and diversity of LLMs supplies a support-based variant: it asks what fraction of original-problem traces fall inside the nearest-neighbor support of twin traces, and that paper's "instruction-tuning tax" is exactly the recall collapse we expect memorization to cause. That work needed about 4,000 samples to stabilize its corpus-level estimates. Our per-item sets are small (k = 16), so we use energy distance per item and reserve precision–recall estimates for benchmark-level aggregates. The encoder is a known confound. We therefore embed traces after masking digits, compute every score under two encoders, and report how much the scores agree.
Curation application. DART-Math curates by spending more sampling budget on hard queries. Fine-tuning Llama-3-8B on its data lifts MATH accuracy from 21.2% to 46.6%. Reasoning Quality Emerges Early curates with TEMP, which scores early-token loss sensitivity under weight noise. Neither paper checks whether its gains survive on uncontaminated variants. We ask two questions. First, how much of each curated model's gain over a random subset of equal size survives on twins? Second, do these selectors enrich contaminated items when benchmark problems are planted in the candidate pool?
Baselines
The do-nothing reference is the raw benchmark score with no audit, together with chance-level AUROC (50%) for separating seen from unseen items.
The likelihood detectors are Loss, Min-K% Prob and LiRA, run exactly as in On The Fragility of Benchmark Contamination Detection in Reasoning Models, on our replication of its Stage I and Stage II pipelines with identical member and non-member splits. The reported reference points are these. LiRA reaches 89.1% AUROC after SFT and drops about 14 points after one clean GRPO epoch. Reference-free detectors fall to about 50%. In Stage II, all ten detectors reach only 50–58% AUROC despite an 11.7% score inflation. Beating this family means our score clears 75% AUROC in the post-GRPO Stage I setting and clearly exceeds 58% in Stage II.
The strongest existing behavioral method is the partial-prompt completion audit of Reasoning or Memorization?, which truncates problems at 40/60/80% and checks whether the model still answers correctly. The file reports that Qwen2.5-Math-7B reaches nearly 40% accuracy on MATH-500 from 40% prefixes. No reported per-item AUROC was found on Lacuna for this audit under GRPO cloaking. We will run it on the same models with the same number of samples per item.
The baseline that isolates our key ingredient is the twin accuracy gap: the same twins and the same k samples, but scoring only correct-on-original minus mean correct-on-twins, with no trace geometry. No reported result was found on Lacuna for this per-item variant. We add two further ablations. One removes the twin–twin null term. The other replaces energy distance with a scalar dispersion measure (mean pairwise trace similarity on the original alone). The distributional claim holds only if the full score beats the accuracy gap by at least 5 AUROC points at equal sample budget. Otherwise we report that the twins, not the distance, carry the signal.
Why not just…?
Why not just use partial-prompt completion? It is cheap and does not rely on likelihoods, so GRPO's variance compression should not touch it. But it only fires when the problem statement itself was memorized verbatim. Contamination introduced as solution traces during fine-tuning, as in Stage II, need not produce prefix recall. Our experiments test this directly on the same checkpoints. If prefix completion matches our score in Stage II, the cheaper method wins and we will say so.
Why not just measure the accuracy drop on perturbed variants? This is the real threat. At the benchmark level, aggregate gaps like those in BeyondBench may well suffice. Per item, though, k binary outcomes are coarse. The accuracy gap also misses cases where the twin answer is right but the trace copies the original's intermediate values. The matched-budget ablation above settles which reading is correct, and we report sample-efficiency curves over k.
Why not just compare output distributions to a clean model with MAUVE? An unpaired comparison mixes contamination with the style shift that RL and instruction tuning cause anyway. Exploring Precision and Recall to assess the quality and diversity of LLMs documents that recall collapse occurs without any leakage. Pairing each problem with its own twins removes this confound. We include the unpaired variant as an ablation and expect it to flag RL-tuned but clean models as contaminated.
Why not just switch to fresh benchmarks like BeyondBench or RandomCalculation? They prevent future leakage but cannot audit existing scores on MATH or AIME. They also measure a different construct: BeyondBench's finding that math fine-tuning hurts algorithmic tasks shows the two do not substitute. We use procedurally generated RandomCalculation items only as a never-seen control for false positives.
Proposed Experiments
Controlled detection. We replicate both contamination stages of the Fragility paper: Qwen2.5-7B-Instruct contaminated during SFT then trained with clean GRPO, and DeepSeek-R1-Distill-Llama-8B contaminated with chain-of-thought traces. Half of each benchmark's numeric items are members. We sample k = 16 traces per original and per twin, and report AUROC for every detector after SFT and after each GRPO epoch. The decisive pattern is this: the likelihood detectors fall toward 50% across GRPO epochs while our score stays roughly flat.
Null calibration. On Llama3.1-8B, and on RandomCalculation items for every model, the score must stay centered at zero with a false-positive rate at our threshold below 10%. Twins whose answers fail verification, or that change the clean model's accuracy, are dropped and counted.
Transfer versus memorization. In Stage II, we split members by whether twin accuracy is also inflated relative to non-members. Our prediction is that the score fires mainly on non-transferring members. If it fires equally on both groups, the "validity-relevant contamination" interpretation fails.
Curation audit. We fine-tune Llama-3-8B on DART-Math-Hard, DART-Math-Uniform, a TEMP-selected OpenThoughts-Math subset, and random subsets of equal size. For each, we report original versus twin accuracy and the benchmark-level score. We then plant 500 benchmark problems with solutions into the candidate pools. Measuring how many survive each selector shows whether difficulty- or loss-sensitivity-based curation concentrates leaked data.
Across all experiments, a result counts as decisive only if it holds under both encoders and with digits masked.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
- Exploring Precision and Recall to assess the quality and diversity of LLMs
- DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving
- Reasoning Quality Emerges Early: Data Curation for Reasoning Models