When a language model solves a benchmark problem it has already seen in training, its written reasoning can look flawless while doing no work. The final answer comes from memory, and the steps are decoration. Current contamination detectors look for this memory in the model's token probabilities, but reinforcement-learning post-training erases that signal: after one epoch of GRPO the gap between seen and unseen items largely disappears, and ten detectors fall to 50–58% AUROC on reasoning models (On The Fragility of Benchmark Contamination Detection in Reasoning Models). This proposal tests behavior instead of probabilities. We edit an intermediate step of the model's own reasoning, for example by restating a problem quantity with a different value, and let the model continue. A model that is actually computing should carry the edit through to a different, predictable answer, or say openly that it noticed the discrepancy. A model replaying a memorized solution tends to return the original answer without mentioning the edit. We call this silent reversion, and we propose it as an instance-level, text-only contamination signal that should survive probability cloaking.
Causal decomposition makes this efficient and tells us where the memorization sits. We treat the reasoning trace as a causal chain in which each step should depend on the steps before it. Memorization then acts as a hidden common cause behind many steps at once, the situation that confounded components are defined to describe (A General Identification Condition for Causal Effects). Separator-based recursive splitting, borrowed from causal discovery, locates the memorized span with a logarithmic number of edits rather than one edit per step. An LLM judge sorts each continuation into one of three outcomes: it propagates the edit, it corrects the edit and says so, or it reverts silently. The judge's reliability is checked against exact arithmetic ground truth.
Core Claim
For a question $q$ with reasoning trace $s_1,\dots,s_K$, we replace step $s_k$ with a counterfactual $s_k'$ whose downstream answer $a'$ is computable exactly. We then sample $n$ continuations. The per-step silent-reversion rate is
$$ \rho_k = \frac{1}{n}\sum_{i=1}^{n} \mathbb{1}\left[\hat a_i = a_{\text{orig}} \ \wedge\ \text{no acknowledgment of the edit}\right], $$
where $\hat a_i$ is the answer of continuation $i$, $a_{\text{orig}}$ is the answer to the unedited question, and acknowledgment is judged from the continuation text. The item-level score is the maximum of $\rho_k$ over tested steps.
Our claim is that this score separates contaminated from clean items at the instance level. We expect it to do so even after GRPO has pushed log-probability detectors to chance. The reason is that GRPO's clipping compresses likelihood gaps, but nothing in that paper suggests it teaches a memorizing policy to depend on its own intermediate steps.
The three-way split is what separates this signal from simply having a robust model. Honest self-correction is legitimate, because the original question stays in context. What marks memory is correction that happens silently. This parallels the finding in The Silent Judge that models act on cues their rationales never mention: across all its tests, justifications cited the injected cue 0% of the time.
Research Program
The causal picture is a chain $q \to s_1 \to \dots \to s_K \to a$. A latent variable $M$, the item's memorized solution, may point into several steps and into the answer. In semi-Markovian terms, $M$ adds bidirected edges, and the steps it touches fall into one confounded component with the answer.
Two features distinguish this setting from classical causal decomposition. First, because we can overwrite text, we can physically intervene on any step. We therefore do not need do-calculus identification of the kind the ID algorithm settles (Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models). The confounded component is our definition of the target, the memorized span, not an estimation device. Second, the practical cost is the number of interventions, which is where recursive decomposition earns its place.
Following the cut-and-recurse logic of SADA and DiCoLa, and the binary-search use of small separators in Meek Separators, we intervene on a middle step and recurse:
- If downstream steps respond as a correct computation would, the hidden memory does not span that cut, and we recurse into the upstream half.
- If they revert silently, we recurse into both halves to find the earliest step whose edit is ignored.
These frameworks learn graph structure from observational data. We reuse only their idea of splitting at a separator, applied to a chain where interventions are free but each one costs $n$ model samples.
The method has three parts.
Counterfactual edit generator. Edits are built from programmatic templates on GSM8K and MSCalc, so the edited answer $a'$ is exact. MSCalc uses exact rational arithmetic with controlled chain-of-thought length (A Narrowing Geometry in Contaminated Reasoning). For AIME and AMC, a verifier computes $a'$ where possible and the item is discarded otherwise.
Three-way outcome judge. The judge is prompted to classify each continuation as propagate, acknowledged correction, or silent reversion. It is calibrated against exact labels on MSCalc before being trusted on free-form items.
Recursive search. This is the separator procedure above.
The method needs the ability to prefill the model's reasoning. That is trivial for open-weight models and available on some commercial interfaces. This is a real restriction on scope, and we state it.
As a secondary question, we ask whether AgentEval-style step judges (AgentEval) score memorized traces as sound. AgentEval assumes step failures are endogenous to the trace and has no notion of a hidden driver. If step judges rate silently reverting traces as high quality, that confirms a blind spot in step-level judging, not just in detectors.
Baselines
Naive reference. Benchmark accuracy with no audit, which cannot tell contaminated from clean items, and a random detector at 50% AUROC. Every method is scored by instance-level AUROC against known contamination labels.
Strongest existing detector: LiRA and the other probability-based detectors. Following On The Fragility of Benchmark Contamination Detection in Reasoning Models, this group includes Min-k%, raw loss, and LiRA. That paper reports LiRA at 89.1% AUROC after supervised fine-tuning. One GRPO epoch reduces LiRA by about 14% and loss by about 16%, and the ten detectors average 50–58% AUROC after long chain-of-thought training. We rerun them on exactly our contaminated checkpoints with white-box access. Our method sees only text and gets no tuning beyond the judge calibration set.
We count as beating this baseline if, after GRPO, silent reversion reaches at least 70% AUROC while these detectors sit in the 50–58% band. We also require that it not trail LiRA by more than a few points before RL.
Paired-confidence test. PaCoST is included because it is the rephrasing-based alternative. It gives only dataset-level verdicts, so we compare at the dataset level: the fraction of contaminated splits flagged at $p<0.05$.
Analogical variants. The A-SCS variants of Beyond Benchmarks swap numbers, insert distractors, and reverse conditions in the question itself. We score each item by its accuracy drop from original to variant, using the same templates and the same sampling budget per item as our method.
Direct judge query. A GPT-4o-class judge reads the trace and is asked whether it looks memorized. This is run with the same judge model we use for outcome classification.
Ablations isolate our key ingredients:
- Editing only the final step, which removes localization.
- Counting any answer reversion as a signal, which drops the acknowledgment distinction.
- Exhaustive per-step editing versus recursive search, compared on both AUROC and number of model calls.
Why not just…?
Why not just perturb the question, as A-SCS does? This is the most serious competitor, and it may match us on detection. The case against it is conceptual but testable. The Fragility paper argues that chain-of-thought-trained models internalize the reasoning pattern of leaked items. If so, they should transfer to number-swapped variants and show little accuracy drop. Our method instead asks whether the model's answer depends on its own steps, which is the property contamination breaks. Question perturbation also cannot say which step is memorized.
The head-to-head AUROC comparison after GRPO settles this. If variant accuracy drop matches silent reversion, we will report the simpler method as sufficient for detection and keep our method only for step localization.
Why not just ask the judge whether the trace is memorized? The file gives two reasons to expect failure. Preference Leakage finds judges near chance at recognizing outputs of their own students, even while rewarding them. The Silent Judge shows rationales do not surface the shortcuts that drive verdicts. We still run this baseline, because a positive result would make the whole intervention machinery unnecessary.
Why not just use white-box detectors with a better threshold? Rethresholding does not recover a signal whose gap clipping has compressed. The Fragility paper shows the drop across all ten detectors, not a calibration shift. The activation-subspace method of A Narrowing Geometry in Contaminated Reasoning is an interesting white-box alternative, but it restores reasoning on only about half of leaked samples. It has so far been shown only on linear MLP projections and arithmetic tasks. We include its eigenspectrum concentration index as an extra white-box detector where compute allows.
Proposed Experiments
Controlled contamination. We follow the Fragility protocol. An open reasoning model, for example a DeepSeek-R1-Distill-Llama-8B-scale model, is fine-tuned with a random half of GSM8K, MSCalc, AMC, and AIME test items mixed into its training data. It is then given GRPO on clean data. This yields three checkpoints (before contamination, after SFT, after GRPO) with exact per-item labels.
Decisive check. The central test is AUROC at the GRPO checkpoint. We require silent reversion to be well above the 50–58% band while LiRA and loss are inside it. A second requirement is that the clean base model shows a low silent-reversion rate on the same items. Without that, the signal would mean question familiarity from pre-training rather than our planted leak.
Judge reliability. On MSCalc we measure agreement between judged outcomes and exact labels. A low agreement rate forces fallback to symbolic answer checking plus a narrow acknowledgment classifier.
Localization and cost. We plant contamination in partial solutions: only the first $j$ steps of a leaked solution are trained on. We then check whether recursive search recovers $j$ with roughly $\log K$ edits versus $K$ for exhaustive editing.
Step judges. We score all traces with an AgentEval-style per-step judge and test whether silently reverting traces receive step scores equal to or higher than clean correct traces.
Uncontrolled audit. Finally, we apply the method to public models on AIME '24, where Beyond Benchmarks already found default setups reversed rankings in 9 of 18 pairwise comparisons. Here we report item-level silent-reversion rates without claiming ground truth.
References
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- A General Identification Condition for Causal Effects
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models
- SADA: A General Framework to Support Robust Causation Discovery
- A Recursive Decomposition Framework for Causal Structure Learning in the Presence of Latent Variables
- Meek Separators and Their Applications in Targeted Causal Discovery
- A Narrowing Geometry in Contaminated Reasoning
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- PaCoST: Paired Confidence Significance Testing for Benchmark Contamination Detection in Large Language Models
- Beyond Benchmarks: Toward Causally Faithful Evaluation of Large Language Models
- Preference Leakage: A Contamination Problem in LLM-as-a-judge