A model that has trained on a benchmark problem tends to have been pulled into a fixed way of solving it. Push it off that path by one token and it steers straight back. On a problem it has never seen, the same push sends it somewhere new. We propose to measure this "steering back" directly. The procedure generates one solution, perturbs it at a few uncertain positions, and scores how likely the model still finds the original continuation. All perturbed branches are packed into a single forward pass using a tree-shaped attention mask, so the cost is one rollout plus one scoring pass. The existing behavioral detector samples 32 full rollouts per question.
This matters because the cheap detectors have stopped working. Contamination detectors that read token probabilities of benchmark text drop to chance once a model goes through reinforcement learning (RL) post-training. The detectors that still work do so by sampling many reasoning traces, which costs seconds per question and makes routine auditing of large benchmark suites impractical. If the attractor strength of a single trajectory carries the same signal as the spread of 32 samples, contamination auditing of reasoning models becomes about as cheap as computing perplexity again.
Core Claim
Contamination introduced by supervised fine-tuning (SFT) or by RL with verifiable rewards leaves a prompt-specific basin of attraction around the model's reasoning trajectory. The width of that basin can be read out by teacher-forced scoring of perturbed branches, without sampling.
Concretely, let $q$ be a benchmark prompt and $y$ the model's greedy solution of length $T$. Choose $M$ fork positions $t_i$ where the model's next-token entropy along $y$ is highest. At each fork, replace $y_{t_i}$ with the runner-up token $\tilde y_{t_i}$. Then score the next $L$ original tokens under both the perturbed and the unperturbed prefix. The basin score is
$$B(q) = \frac{1}{M L}\sum_{i=1}^{M}\Big[\log p\big(y_{t_i+1:t_i+L}\mid q, y_{<t_i}, \tilde y_{t_i}\big) - \log p\big(y_{t_i+1:t_i+L}\mid q, y_{<t_i}, y_{t_i}\big)\Big].$$
A score near zero means the perturbation costs almost nothing, so the model returns to its original path. A strongly negative score means the perturbation genuinely redirects the reasoning. We predict that $B$ is higher on seen prompts. The same statistic can be anchored on the gold chain-of-thought instead of the model's own rollout, which targets contamination that came from training on reference solutions.
Two properties make this more than a rename of existing scores. First, it scores the post-RL policy's own output rather than benchmark text. The likelihood gap on benchmark text is exactly what RL clipping erases. Second, it measures robustness across the sequence (does an early deviation get corrected later?) rather than whether each individual token is a local peak. That second property distinguishes it from Min-K%++.
Research Program
Evidence that motivates the proposal. On The Fragility of Benchmark Contamination Detection in Reasoning Models contaminates models with SFT and then applies clean GRPO. It shows that importance-ratio clipping shrinks the member/non-member log-probability gap and drives ten likelihood detectors toward chance. The paper diagnoses the failure but proposes no fix.
Detecting RLVR Training Data via Structural Convergence of Reasoning finds that prompts seen during RL produce structurally similar traces. Its Min-$k$NN Distance detector samples 32 completions and averages the smallest pairwise edit distances. It reaches an average AUC of 0.70 at roughly 6.65 seconds per item. Our hypothesis is that trace similarity across samples is the observable consequence of a basin around any single trace. If so, the basin can be measured by scoring rather than sampling.
A Narrowing Geometry in Contaminated Reasoning reports that contaminated reasoning funnels activations through a few dominant directions. This gives a mechanistic reason to expect perturbations to be damped. We use that finding as motivation only; we do not intervene on representations.
Where the method comes from. The mechanism reuses efficient-scoring techniques rather than inventing them. Efficient Autoregressive Inference for Transformer Probabilistic Models encodes shared context once and places targets in a causal buffer, so many joint densities can be evaluated in one pass. d2 designs attention masks that yield exact trajectory likelihoods of a diffusion model in a single pass. Transcormer uses sliding masks to get every token's conditional in one pass.
We adapt the buffer idea to ordinary causal language models. The full trajectory $y$ is the shared context. Each of the $M$ perturbed branches ($\tilde y_{t_i}$ followed by the $L$ original tokens) is appended as a buffer segment. Each segment attends to $q, y_{<t_i}$ and causally to itself, but not to other branches. Both terms of $B$ then come out of one forward pass over roughly $T + M(L+1)$ tokens.
Where the fork selection comes from. Fork selection borrows its logic from JUMP. JUMP replaces random masking with a learned head that picks hard tokens, so that membership inference on fine-tuned diffusion models needs only 2+1 passes. We keep JUMP's insight that the signal lives at uncertain positions. We depart from it in three ways: we target autoregressive reasoning models, we need no reference base model, and we probe with substitution-and-rescore rather than masked reconstruction.
Build and validation plan. We will implement the packed scorer and verify that it matches sequential scoring to numerical precision. We will then sweep $M$ and $L$. Finally, we will test whether $B$ correlates, per prompt, with Min-$k$NN computed from 32 samples. That correlation is the direct test of the hypothesis that one basin predicts many samples.
Baselines
Naive reference points. The first is the loss on the benchmark's gold text. The second is Min-K%++ (Min-K%++), which z-normalizes each token's log-probability against the vocabulary distribution and averages the lowest $k\%$. Both are single-pass and reference-free, which makes them the right cost-matched floor.
The file reports that on these RL-trained models these baselines collapse. After one epoch of GRPO, Loss AUROC drops by about 16 points and moves toward roughly 50%, while LiRA falls from 89.1% by about 14 points. In the Stage II setting on DeepSeek-R1-Distill-Llama-8B, detectors average 50–58% AUROC; all of these figures come from the fragility paper. We will rerun both baselines on our own checkpoints rather than rely on these numbers. The goal is to confirm the collapse under identical data.
Strongest existing method. This is Min-$k$NN Distance from the structural-convergence paper. It reports an average AUC of 0.70 on RL-trained reasoning models from 1.5B to 32B parameters, 0.76 on OpenR1-Distill-7B, and robustness to paraphrase (0.72 to 0.71). No reported result was found on Lacuna for Min-$k$NN on the fragility paper's SFT-then-GRPO or Stage II settings, so we will run it there ourselves.
We run Min-$k$NN at its published $N=32$, which gives it a large compute advantage. We also run it at a budget matched to ours (about $N=2$ to $4$ rollouts, with the scoring pass counted in rollout-equivalents of wall-clock time). Both variants use the same sampling temperature and the same prompts. Tuning is limited to $k$, and we tune $k$ on a held-out split exactly as we tune $M$ and $L$.
Counting as a win. Beating the matched-budget Min-$k$NN by a clear AUC margin on every setting counts as a success. So does matching $N=32$ Min-$k$NN within its seed variance at a tenth or less of the wall-clock cost.
Ablations that isolate the key ingredient. Two ablations test whether the perturbation itself carries the signal. The "no-perturbation" ablation keeps only the second term of $B$, the likelihood of the model's own continuation; this is Min-K%++-style scoring on self-generated text. The "entropy-only" ablation uses the mean or top-$M$ next-token entropy along the greedy path. A third ablation replaces entropy-chosen forks with random positions.
Why not just…?
Why not just use the token entropy of one greedy rollout? This is the most serious threat. RL is known to sharpen policies, and low entropy along a single trace is the cheapest possible proxy for low sampling diversity. Our reason to expect it to fall short is that GRPO sharpens the policy on unseen prompts too. The fragility paper's mechanism, in which clipping disproportionately changes non-member statistics, suggests that marginal, position-wise confidence is exactly what gets equalized. Basin score instead asks whether the model corrects a deviation several tokens later, which is a property of the prompt-specific trajectory rather than of global sharpness.
That argument is a hypothesis, not evidence, so the entropy-only ablation is a headline comparison and not a footnote. If entropy matches $B$ within noise across settings, we will report that the cheap proxy suffices. The contribution would then shrink to showing that single-rollout statistics recover the Min-$k$NN signal.
Why not just run Min-$k$NN with fewer samples? Edit distances between a handful of long traces are high-variance. Min-$k$NN's own design relies on nearest neighbors among 32 samples. Whether it degrades gracefully at $N=4$ is not reported in the file, so the budget-matched run settles it directly.
Why not just apply Min-K%++ or a reference model such as LiRA? Both read the gap in likelihood on benchmark text, and the fragility paper shows that RL clipping erases this gap. Reference-model methods also need a matched clean model. ConStat fits against 20 reference models and detects inflated, non-generalizing performance at the benchmark level rather than per item. Basin scoring is reference-free and per-prompt, and we include Min-K%++ and LiRA as baselines to confirm the gap.
Why not just use n-gram overlap or partial-prompt completion? Training corpora are usually undisclosed, and verbatim matching was near chance in the fragility study. Partial-prompt completion from Reasoning or Memorization? works when contamination comes from pre-training: Qwen2.5 completes MATH-500 at 65.8% exact match from an 80% prefix. However, it needs full generations and targets verbatim recall rather than the reasoning-level contamination that CoT training produces. We use it as a sanity check on Qwen2.5, not as a competitor.
Proposed Experiments
Settings. We use three settings, all with labeled members and non-members. The first reproduces the fragility paper's Stage I recipe: SFT on a known half of AIME, AMC, OlympiadBench, and Minerva Math, followed by clean GRPO, with basin scores tracked across GRPO epochs. The second is the Stage II setting, which injects CoT contamination into DeepSeek-R1-Distill-Llama-8B. We evaluate $B$ anchored on the model's own rollout and $B$ anchored on the gold CoT. The third uses the RL-training-data setting of the structural-convergence paper at the 1.5B–7B scales, including its distillation case.
Metrics. We report AUROC, TPR at 5% FPR, and wall-clock time per item. RandomCalculation from Reasoning or Memorization? serves as a guaranteed-clean control: a working detector should not flag it.
Decisive checks. Three results decide the outcome. The first is whether $B$ keeps its AUROC across GRPO epochs while Loss and Min-K%++ decay. This is the direct test against cloaking. The second is whether $B$ beats both the entropy-only and the no-perturbation ablations. If it does not, the perturbation adds nothing and the claim fails. The third is whether the per-prompt correlation between $B$ and 32-sample Min-$k$NN is substantial. If it is, the one-basin-predicts-many-samples mechanism is supported; if $B$ works but does not correlate, it is detecting something else, and we will say so.
A final stress test paraphrases member prompts, following the structural-convergence paper's paraphrase protocol. It checks whether the basin follows the problem's structure or its surface wording.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Detecting RLVR Training Data via Structural Convergence of Reasoning
- A Narrowing Geometry in Contaminated Reasoning
- Efficient Autoregressive Inference for Transformer Probabilistic Models
- d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation
- Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
- JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- ConStat: Performance-Based Contamination Detection in Large Language Models
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination