When a language model has seen a test question during training, its high score may reflect recall rather than reasoning. For modern reasoning models, the standard contamination tests fail. Those tests flag a question as "seen" when the model assigns it unusually high probability. Reinforcement learning (RL) post-training (fine-tuning the model with reward signals rather than next-token likelihood) erases that probability gap. After it, the standard detectors are close to coin flips.
We propose a different kind of test. Let the model solve a problem once. Then make it solve the problem again with a decoding-time penalty that discourages it from reusing phrases from its first solution. The penalty is a small subtraction from the scores of tokens that would continue a phrase already present in that first attempt. A model that actually reasons should find another route to the same answer at modest cost. A model that memorized one solution path should either resist the detour, drifting back to the same path in paraphrase, or keep the memorized answer while its new derivation no longer supports it. The contamination score is the model's response to a controlled, graded push off its preferred path.
Core Claim
Repetition penalties were built to stop models from looping. Two recent designs let them preserve necessary structure. The first is suffix-aware penalization with "sequence breakers" that exempt formatting tokens such as newlines and colons (Don't Repeat Yourself). The second is a compression-based penalty that subtracts from each candidate token's score in proportion to how cheaply an LZ77 compressor would encode it (LZ Penalty). Both papers apply the penalty to the text generated in the current pass.
Our change is to point the penalty at a reference trace: the model's own first, deterministic (greedy) solution. For the LZ variant, this means pre-filling the compressor's dictionary with that trace. Tokens that would re-enter the first path then become cheap to encode and are penalized.
The claim is that the dose-response curve of this cross-trace penalty separates contaminated from clean items after RL post-training, where likelihood detectors cannot. Concretely, we sweep the penalty strength α and record two quantities at each α. The first is how far the new trace actually diverges from the reference. The second is whether the final answer survives, and whether the new derivation still supports it.
Why this, and why now
Three findings motivate the probe.
- RL contamination produces "policy collapse". Detecting Data Contamination from RL Post-training shows that RL contamination narrows the model onto a single deterministic reasoning path. Their Self-Critique probe prompts the model for an alternative solution and compares the token-entropy sequences of the two responses.
- Likelihood detectors collapse on reasoning models. On The Fragility of Benchmark Contamination Detection in Reasoning Models proves that clean GRPO training erases the likelihood gap. It also shows that contaminating models with chain-of-thought traces pushes ten detectors down to 50–58% AUROC.
- Memorized answers can override the model's own reasoning. Deciphering the Factors Influencing the Efficacy of Chain-of-Thought documents models producing valid reasoning tokens but emitting a memorized high-probability answer at the end. In the same vein, STIM attributes 67.3% of GSM8K reasoning errors to short-range "local memorization".
Together these suggest that contamination lives in the coupling between path and answer, not in raw likelihood.
Self-Critique is the closest existing work, and we differ from it in three ways.
- The perturbation is at the logit level, not a prompt. A request for "another approach" can be satisfied by cosmetic rewording. A penalty on reused phrases cannot, and it is not confounded by how the model interprets the instruction.
- The perturbation is graded. We read a curve over α rather than a single similarity number.
- We score the answer and its support. Self-Critique compares entropy profiles only. We check whether the answer persists and whether the forced-detour derivation actually justifies it, using the step-level verifier VersaPRM (a process reward model that grades individual reasoning steps, as used in STIM).
The penalty papers constrain the design. DRY shows that naive penalties hurt reasoning on GSM8K and MMLU by punishing reused variable names and syntax. We therefore inherit its sequence breakers and minimum match length, so that notation can repeat and only multi-token paths are blocked.
Research Program
Build the probe. We implement two cross-trace penalties.
- Cross-trace DRY. Suffix matching runs against the concatenation of the reference trace and the current context. The penalty takes DRY's exponential form, λβ^(n−L), where n is the match length and L is the minimum length before penalization begins.
- Cross-trace LZ. The penalty takes the form ℓ ← ℓ + αΔ|C|, where ℓ is a token's score and Δ|C| is its marginal codelength under an LZ77 compressor whose window is pre-filled with the reference trace.
For each item we run one greedy reference pass, then penalized greedy passes at a small grid of α values. The main grid has four values; we also report a two-pass version that uses a single α, to compute-match Self-Critique.
Read out. At each α we record three quantities:
- trace divergence, measured as the fraction of 4-grams in the new trace that also appear in the reference;
- answer agreement with the reference;
- VersaPRM's verdict on whether the new derivation supports its own answer.
From these we define two item-level statistics.
- Resistance is the divergence the model achieves per unit α. We expect low values when the model paraphrases its way back onto the memorized path.
- Unsupported persistence is the rate at which the answer survives while the derivation fails verification.
We do not assume in advance which direction contamination moves each statistic. Signs and a simple two-feature logistic combination are fit on a calibration split with known membership, then frozen before any test split is scored.
Analyze mechanism. Using token-level attributions in the style of STIM, we check where penalized traces re-merge with the reference. If contaminated items re-merge at problem-specific phrases (numbers, intermediate results), the probe is exposing memorized content. If they re-merge only at generic framing phrases, the signal is stylistic and weaker.
Baselines
Naive reference. The do-nothing reference is perplexity and Min-K% (the average log-probability of the least likely K% of tokens) on the question and the greedy trace. These are cheap and reference-free. On the RL-MIA benchmark they are reported at 0.42–0.53 AUC (Detecting Data Contamination from RL Post-training). We rerun them on identical items and models so that our comparisons are internal, not only cross-paper.
Strongest existing method. The strongest competitor is Self-Critique from the same paper. It is the only method in the file that works after RL post-training and targets the same phenomenon, policy collapse. The reported average AUC is 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct. We run it with its published prompt on the same models and items. We give it the same total number of decoding passes as our probe: either our two-pass variant against its native two passes, or our four-α sweep against Self-Critique averaged over four alternative-path prompts. Beating it means a higher AUC on both models under matched passes, with a paired bootstrap interval over items that excludes zero, plus a higher true-positive rate at 5% false-positive rate. On the Fragility paper's controlled Stage II setting, beating the reported 50–58% AUROC by a clear margin would be the more important result, since no detector there currently works.
Temperature-sweep control. This baseline replaces the targeted penalty with temperature sampling. We calibrate each temperature to produce the same average 4-gram divergence from the reference as the corresponding α, then compute the same answer-persistence and PRM statistics. It tests whether any divergence-matched perturbation suffices.
Ablation isolating the key ingredient. The ablation keeps the entire pipeline but fills the penalty history with the greedy trace of a different, randomly chosen problem of similar length. This preserves the generic cost of penalized decoding while removing the targeting at the model's own path. If this "wrong-trace" probe matches the real one, our claim fails.
A second ablation removes DRY's sequence breakers. It checks that the signal is not an artifact of damaging formatting on some items more than others.
Why not just…?
Why not just sample at higher temperature and measure answer consistency? This is the most serious threat. Self-consistency-style agreement is cheap and could plausibly track policy collapse on its own. The file offers only indirect evidence against it: LZ Penalty reports that reasoning models perform best near temperature zero. That suggests temperature degrades clean items as well, which would blur the separation. Temperature also perturbs every token uniformly, including tokens that do not lie on the memorized path. The divergence-matched temperature baseline settles the question directly. If it reaches our AUC, we will report that the targeting adds nothing and that the finding reduces to consistency under perturbation.
Why not just ask the model for an alternative solution, as Self-Critique does? A prompt tells the model what to do but does not force it. A policy-collapsed model can comply by rewording the same derivation. Self-Critique's reliance on entropy-profile similarity is partly a workaround for this. Whether a hard constraint beats a soft request is exactly what the compute-matched comparison measures. We will also report the fraction of Self-Critique "alternatives" that share more than half their 4-grams with the original, as a direct measure of cosmetic compliance.
Why not just use perplexity or Min-K%? The theory and experiments in On The Fragility of Benchmark Contamination Detection show that GRPO clipping removes the likelihood gap these detectors depend on. We include them only as the naive floor.
Why not just switch to dynamic benchmarks such as LiveBench? Fresh benchmarks protect future evaluations. They cannot audit existing models' scores on AIME or GPQA, and those historical scores are exactly what needs auditing. Our probe is complementary. Post-cutoff LiveBench items (LiveBench) give us a supply of true non-members for calibration.
Proposed Experiments
Natural RL contamination. We use RL-MIA (AIME 2024, AIME 2025, Knights & Knaves, SAT) on Qwen2.5-7B-Instruct and DeepSeek-Math-7B-Instruct, where Self-Critique's numbers are reported.
Controlled contamination. Following On The Fragility of Benchmark Contamination Detection, we construct both stages ourselves on AIME, AMC, OlympiadBench, Minerva Math, and GPQA Diamond.
- Stage I: fine-tune on leaked items, then run clean GRPO.
- Stage II: contaminate with chain-of-thought traces.
That paper reports that leaked-solution fine-tuning inflates pass@1 by 8.8–11.7%. We verify that our inflation lands in that range before trusting the construction. Membership is known by design.
Metrics. We report AUROC, TPR at 5% FPR, and pass count per item. Each result is stratified by whether the reference answer was correct, so the probe is not simply detecting easy items.
Decisive checks. Three results would kill the idea:
- the wrong-trace ablation matches the real probe;
- the divergence-matched temperature baseline matches it;
- in Stage II, the probe's AUROC stays inside the reported 50–58% band.
Conversely, the probe earns its place if three things hold. It must beat Self-Critique under matched passes on RL-MIA. It must clear the Stage II band. And the mechanism analysis must show that contaminated items re-merge with the reference at problem-specific content rather than generic phrasing.
Sanity check. We confirm that penalized decoding on clean items keeps AIME and GPQA accuracy near the unpenalized level, consistent with the LZ Penalty paper's report that its penalty maintained those scores. If it does not, the probe is measuring general damage rather than path dependence.
References
- Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time
- LZ Penalty: An information-theoretic repetition penalty for autoregressive language models.
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
- Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark