When a language model has seen a benchmark's test questions during training, its score on those questions goes up. The open question is whether that extra score survives when the same question is restated in a different "style": new numbers, new names, a different register, or an invented notation. Generative domain augmentation, a line of work in computer vision, has spent a decade learning to change an input's style while keeping its meaning fixed. This proposal borrows that idea to build an audit. We take benchmark items and fresh control items of the same kind, push both through the same ladder of increasingly distant meaning-preserving rewrites, and measure how fast accuracy decays on each. A clean model loses accuracy at the same rate on both sets. A contaminated model has an advantage on benchmark items at the original wording that shrinks as the rewrites move further away, and the control set has no such advantage.
The audit needs only sampled answers, not token probabilities. That matters because the probability-based signals current detectors rely on are exactly what modern post-training erases. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows two things. First, clean reinforcement learning (RL) after contaminated supervised fine-tuning keeps the score gain but collapses the loss gap between seen and unseen items. Second, training on worked chain-of-thought solutions pushes all ten tested detectors to 50–58% AUROC. If the score gain survives post-training, a detector should look at the score gain directly, and look at it in a way that separates "memorized this item" from "this item is easy" and from "the rewrite made it easier."
Core Claim
Contamination should be measured as an interaction rather than a single difference. Let $a_i(k)$ be the model's mean accuracy over $S$ samples on item $i$ after a rewrite at distance level $k$, with $k=0$ the original wording. Let $B$ be the audited benchmark items and $C$ be control items drawn from the same distribution that the model cannot have seen. The audit statistic is
$$ D(k) = \mathbb{E}_{i \in B}\big[a_i(0) - a_i(k)\big] - \mathbb{E}_{j \in C}\big[a_j(0) - a_j(k)\big], $$
computed after matching items in $B$ and $C$ on their original accuracy $a(0)$. The matching removes the ceiling effect, in which contaminated items have more room to fall.
The single difference used by prior perturbation audits is $a(0) - a(k)$ on benchmark items alone. Two nuisances corrupt it: whether the rewrite generator makes questions harder or easier, and baseline difficulty. $D(k)$ cancels both. Plotting $D(k)$ against $k$ also gives a "transfer radius": how far the contaminated advantage reaches before it disappears.
The radius is what lets us answer the question the Fragility paper leaves open. When chain-of-thought contamination makes a model generalize to similar unseen items, is that memorization or learning? Our answer is that it is memorization to the extent that the advantage is confined to rewrites near the training form, and learning to the extent that it persists far out on the ladder.
Why domain augmentation supplies the mechanism
The vision literature supplies three things this audit reuses. It separates semantic content (object identity, layout) from style (texture, medium, weather). It verifies that labels survive the transfer. And it organizes shifts from shallow to deep.
Fourier Domain Adaptation, as used in Pulling Target to Source, swaps only the low-frequency appearance of an image and keeps its segmentation masks. The domain-transformation network in Deep Domain-Adversarial Image Generation for Domain Generalisation learns larger shifts while a classifier enforces label preservation. Style-transfer datasets such as StyleCOCO (Kadish et al.) move whole datasets into new depictions. These works are where the method comes from.
We use their structure, a graded set of style shifts with a content check at every level, in place of their goal, which is building invariant models. Our ladder for math and science reasoning has five rungs:
- Surface rephrasing.
- Entity and variable renaming.
- Re-parameterizing numeric constants, with the gold answer recomputed by a symbolic solver or executed program, in the spirit of DyVal.
- Register change, from word problem to formal statement and back.
- Restatement in an invented notation defined inside the prompt.
For knowledge multiple-choice questions the deep rungs are different. They are option rewriting, translation, and the "None of the other choices" substitution of MMLU-CF. At every rung, content is checked by recomputing the answer where the item is executable, or by an LLM judge plus a human audit sample otherwise.
The motivating evidence comes from the contamination literature. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples shows that fine-tuning on paraphrased test items lifts Llama-2-13B on MMLU from about 55 to about 86, so memorization already transfers across one rung of rephrasing. That is why a single rung is not enough and a ladder is needed.
The constraint, and the reason a control set is needed, comes from Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations. In that work, clean vision-language models gained +31.5% on MMStar after diffusion edits because the generated images were clearer, while contaminated ones fell as far as −45%. That audit therefore depends on the sign of a delta that the generator itself shifts. A generator that made questions slightly harder would have hidden contamination.
Research Program
The work has three parts.
Build the ladder. We first build the rewrite ladder for GSM8K, MATH, AMC and AIME-style problems, and for MMLU. Each rung gets a content-verification step, and we report verification rates per rung. Meta Probing Agents achieved 94% equivalence and 97% correctness under human audit for shallow probes. We expect the deep rungs to be harder to keep faithful and will report this honestly.
Validate under ground-truth contamination. We reproduce the controlled contamination protocols of the Fragility paper, so that membership is known per item. Protocol A, the RL-cloaking case, contaminates Qwen2.5-7B-Instruct by supervised fine-tuning on half of the competition-math items, then runs clean GRPO. Protocol B, the chain-of-thought case, fine-tunes DeepSeek-R1-Distill-Llama-8B on worked solutions to half the items. For MMLU we follow Yang et al.'s rephrased-sample contamination of Llama-2-13B. The unseen half of each benchmark serves as $C$.
Apply to released models. We then apply the audit to public models, using MMLU-CF's closed test set and post-cutoff items as controls. A finding here is only interpretable after the validation stage shows that clean models give $D(k)\approx 0$.
We will not train the ladder adversarially, even though a DDAIG-style minimax generator is the obvious import. A generator rewarded for widening the benchmark–control gap on the model being audited can find spurious gaps. A learned ladder is instead tested as an ablation: fitted on calibration models and frozen before it is applied to held-out models.
Baselines
Naive single-rung delta. This is the do-nothing-new reference point: accuracy on the original items minus accuracy after one rephrasing, with no control set. It is the implicit test behind MPA-style collapse reports, where GPT-4-Turbo drops 15.7% on probed MMLU. We run it with the same generator and the same samples as rung 1 of our ladder, so the only difference is the control subtraction and the extra rungs. Beating it means higher item-level AUROC and a near-zero false-positive rate on clean models, where the naive delta should fire whenever rephrasing changes difficulty.
Log-probability detectors. The strongest existing methods are LiRA, Loss, and Min-K%, as evaluated in the Fragility paper. That paper reports LiRA at 89.1% AUROC after contaminated fine-tuning and before RL, a drop of about 14% after one epoch of GRPO, with Loss dropping about 16%. In the chain-of-thought setting all ten detectors fall to 50–58% AUROC. We run these detectors on the same members and non-members with white-box access, which favors them, since our method uses only sampled outputs. Beating them means exceeding LiRA's post-GRPO AUROC in Protocol A and clearly exceeding the 58% ceiling in Protocol B. We pre-register 70% item-level AUROC in Protocol B as the success threshold.
PaCoST. PaCoST is the strongest rephrasing-based statistical audit. It applies a paired t-test on the model's confidence that a given answer is correct, comparing original and rephrased items. We run it with its own rephraser and the same items. It reports model-level significance rather than an accuracy figure; no reported AUROC was found on Lacuna for PaCoST under RL or chain-of-thought contamination. Beating it means detecting Protocol A and B contamination at $p<0.05$ where it does not, while never flagging clean models.
Fresh-set gap. This is the MMLU-CF-style comparison of accuracy on the original benchmark versus accuracy on a fresh, difficulty-matched set. MMLU-CF reports an average decline of about 14% across more than 40 models, and o1 falling from 92.3% to 80.3%. It uses the same control set we use, so the comparison isolates the value of the ladder over the control alone.
Ablations. Three ablations remove one ingredient each: the control subtraction, the deep rungs (keeping only rungs 1–2), and content verification.
The LLM Decontaminator reports F1 of 0.94 to 1.0, but it requires the training corpus. We treat it as an upper reference for the setting with corpus access, not as a competitor.
Why not just…?
Why not just rephrase once and look at the drop? Because the size and sign of that drop depend on the generator as much as on the model. The VLM perturbation result, where clean models rose by 31.5%, demonstrates this directly. Yang et al. show that rephrased memorization also survives one rung of rewriting. The single-rung ablation is the experiment that tests this. If it matches the full ladder's AUROC in Protocols A and B, we will drop the ladder and report only the control subtraction.
Why not just compare against a fresh contamination-free benchmark? This is the real threat at the level of whole models, and MMLU-CF does it well. It has two weaknesses: it cannot say which items are contaminated, and its signal mixes contamination with any leftover difficulty mismatch between the old and new sets. The difference-in-differences cancels mismatches that are constant per item, and the per-item statistic localizes contamination. The fresh-set-gap baseline is run on identical controls. If it matches our model-level detection and nobody needs item-level localization, the honest conclusion is that the ladder adds only the transfer radius.
Why not just fix log-probability detectors for reasoning models? The Fragility paper attributes their failure to RL clipping, which compresses the spread of loss values on unseen items. The score gain persists, so a behavioral signal is the natural target. Log-probability detectors also cannot run on text-only commercial APIs, which PaCoST's design already concedes.
Why not train an adversarial rewrite generator, as DDAIG does for images? Adversarially maximizing the benchmark–control gap on the audited model invites false positives. The learned-ladder ablation, fitted on calibration models and frozen before audit, will show whether learning the ladder adds power without inflating false positives on clean models.
Decisive Checks
The proposal is wrong if any of three things happens.
First, clean models show a $D(k)$ that is significantly different from zero, which would mean matching on original accuracy fails to remove difficulty confounds.
Second, a model fine-tuned on in-distribution training data that contains no test items triggers the audit. Telling contamination apart from ordinary competence on the domain is the central requirement.
Third, AUROC in Protocol B stays within the 50–58% band, which would mean chain-of-thought contamination transfers across even the invented-notation rung. That outcome would itself be informative: it would show the contaminated gain behaves like learned skill rather than recall.
Success means three things: a near-zero false-positive rate on clean and in-distribution-trained models, item-level AUROC above the post-GRPO LiRA level in Protocol A and above 70% in Protocol B, and transfer-radius curves that rank the contamination conditions in their known order of severity.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Pulling Target to Source: A New Perspective on Domain Adaptive Semantic Segmentation
- Deep Domain-Adversarial Image Generation for Domain Generalisation
- Improving Object Detection in Art Images Using Only Style Transfer
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
- Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
- Contamination Detection for VLMs Using Multi‑Modal Semantic Perturbations
- Dynamic Evaluation of Large Language Models by Meta Probing Agents
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models