Catching Memorized Benchmark Answers by Breaking the Reasoning Midway

When a benchmark leaks into a model's training data, the model can score well by recalling answers instead of working them out. The usual detectors look for this by checking whether the model finds benchmark text unusually familiar, for example through lower loss on it. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that this signal largely disappears in modern reasoning models. A single epoch of reinforcement-learning post-training (GRPO, a policy-gradient method with clipping) cuts loss-based detection AUROC by about 16%. When the leaked data includes worked chain-of-thought solutions, detectors average only 50–58% AUROC, which is close to chance, even though the leak raised benchmark scores by 11.7%. The contamination survives, but the familiarity signal detectors rely on does not.

We propose to audit the reasoning process instead of the text. Take a model's own step-by-step solution to a benchmark item, corrupt one intermediate result (say, change a computed value in step 3), and let the model continue from the corrupted prefix. A model that actually reasons has two honest responses: it carries the wrong value forward and reaches a different answer, or it visibly notices and fixes the error. A model that recalls the answer tends to do something else, which we call silent snap-back. Its later steps are inconsistent with the corrupted value, it never corrects the error in writing, and it still lands on the original answer. Our claim is that silent snap-back, measured per item and compared against the same model's behavior on freshly generated items it cannot have seen, detects contamination in post-RL reasoning models where loss-based detectors fail.

Where the idea comes from and what is new

The framing borrows from process evaluation in education. There, teachers stop grading only the final exam and instead check the student at several milestones, favoring problem-solving logic over bare output (Lili Liu et al.). A student who copied an answer can produce it, but cannot defend or adapt the intermediate steps when a milestone is disturbed. We use this only as a design lens. The contamination literature supplies the problem and the tools.

Three lines of work motivate the proposal. The fragility paper shows that loss-based detection collapses in exactly the regime that matters. How Does Unfaithful Reasoning Emerge from Autoregressive Training? defines an intervention test for faithfulness: perturb an intermediate step and see whether the answer follows. It studies this only in small synthetic transformers and never as a contamination tool. It also constrains us, because it shows that clean models can drift into skip-step reasoning too, so snap-back alone could come from ordinary unfaithfulness rather than leakage. DyVal raises the same concern: 33.8% of GPT-4's failures were answers given without supporting reasoning.

The closest related methods work at the whole-problem level. ConStat compares scores on the original benchmark against rephrased or synthetic reference sets. DyCodeEval rewrites a coding problem's story while keeping its algorithm fixed. MR-GSM8K and GAMEBOT score intermediate steps, but they measure reasoning quality, not contamination, and MR-GSM8K is itself a static dataset that can leak. None of these edits the model's own trace midway and asks whether the answer is anchored to the problem statement or to the reasoning.

We also expect an advantage over reference-set methods specifically. The fragility paper reports that chain-of-thought contamination spreads the model's confidence to similar unseen questions. That inflation should reach ConStat's reference sets too and shrink the gap ConStat measures. Our signal is tied to a single item, since snap-back to a specific memorized answer requires having seen that item.

Research Program

Intervention procedure. For each benchmark item, we sample the model's solution and parse it into steps. We then find intermediate quantities that a symbolic checker can verify, such as numbers, algebraic expressions, or selected options. At a randomly chosen verifiable step, we replace the value with a plausible wrong one and force the model to continue generating from that prefix.

Scoring along three dimensions. Each continuation is scored on three things:

  1. Answer persistence: does the final answer still equal the model's original answer?
  2. Downstream consistency: do the later steps actually use the corrupted value, checked symbolically?
  3. Explicit repair: does the trace flag and recompute the corrupted step, checked with a rule-based matcher and audited by a judge model on a sample?

Silent snap-back is an item where the answer persists, the downstream steps are inconsistent, and no repair appears. We repeat this over k interventions per item and produce an item score, the snap-back rate. We then take the difference between that rate and the model's snap-back rate on matched fresh items.

Calibration controls. The fresh items come from generators that cannot have leaked. For each benchmark domain we build templates that match its difficulty. DyVal provides arithmetic and logic graphs with controllable depth, and BeyondBench provides algorithmic tasks with more than 10^15 instances each. These controls set each model's baseline rate of unfaithful but innocent snap-back. That baseline is the main defense against the confound raised by the unfaithful-reasoning paper.

Clean-room validation. Before moving to frontier-scale models, we reproduce the arithmetic expression setup from the unfaithful-reasoning paper. We train small transformers, contaminate a known subset of items, and check two things. First, silent snap-back should separate memorized items from unseen ones. Second, it should do so even for models trained into the skip-step phase, where clean items should show persistence without a sharp item-level gap.

Proposed Experiments

Main testbed. We follow the fragility paper's contamination protocol on AIME, AMC, GPQA Diamond, OlympiadBench, and Minerva Math, using DeepSeek-R1-Distill-Llama-8B and a second open reasoning model. We create three conditions:

  • Stage I: fine-tuning on leaked question–answer pairs.
  • Stage II: fine-tuning on leaked chain-of-thought solutions.
  • Post-RL: each of these followed by one to several epochs of GRPO.

Contaminated and held-out items are split at random within each benchmark, so ground truth is known. The primary metric is item-level AUROC for separating contaminated from held-out items. Secondary metrics are model-level detection, meaning whether the audited model can be told apart from an uncontaminated twin, and how the signal scales with the number of interventions per item.

Transfer to code. As a second domain, we contaminate a small code model on HumanEval and MBPP. For executed programs, we inject corrupted intermediate variable values into the scratchpad trace. This tests whether the signal transfers where DyCodeEval checked only final pass/fail.

Baselines

Naive reference. Raw benchmark accuracy compared with accuracy on the matched fresh generator items, with no item-level detection. This measures how much inflation a trivial model-level check already reveals.

Loss-based membership detectors. Min-K%, LiRA, and loss variance, run on the same models, items, and splits. The fragility paper reports that these fall to 50–58% average AUROC under Stage II contamination and lose about 14% (LiRA) and 16% (loss) AUROC after one GRPO epoch. We re-run them rather than copy numbers, so that the conditions match.

ConStat. This is the strongest competitor. It is black-box, and in its own paper it detected 98.4% of sample-specific contamination in controlled experiments, against a best baseline of 71%. However, those experiments were not on post-RL reasoning models with chain-of-thought leakage. We found no reported result on Lacuna for ConStat in the post-GRPO Stage II setting. We will build its reference sets by rephrasing plus fresh synthetic items, using the same generator calls we allot to our own fresh controls. Both methods will get equal generation budgets per item.

Item-level problem perturbation. Change a number in the problem statement, so the answer should change, and flag items whose answer does not change. This is the most natural simpler relative of our idea.

Key ablation. We score answer persistence alone, without the downstream-consistency and repair dimensions. This isolates whether the multi-dimensional process scoring matters or whether a crude "answer didn't move" test is enough.

Success criterion. Beating the field means Stage II post-RL AUROC of at least 0.75, clearly above the loss detectors' 50–58%, and significantly above ConStat and problem perturbation under an identical budget.

Why not just…?

Why not just use dynamic benchmarks like BeyondBench or DyVal? They measure clean capability going forward, and we use them as controls. They cannot answer whether an existing AIME or GPQA score, or a specific item, was inflated by leakage. They also check only final answers, so they would not show a model snapping back to a recalled result.

Why not just use loss-based detectors? The fragility paper already shows they reach near-chance AUROC after chain-of-thought contamination and RL. Our experiment re-confirms this on identical splits, so the comparison is direct rather than argued.

Why not just use ConStat? This is a real threat. If chain-of-thought contamination does not inflate distributionally similar reference items much, ConStat may match us at the model level. The decisive comparison is Stage II post-RL item-level AUROC under equal budgets. If ConStat ties us, our remaining contribution is item-level localization plus a readable failure trace, and we will say so.

Why not just perturb the problem statement instead of the reasoning? A model that internalized a solution procedure may adapt correctly to a changed number, which would make perturbation miss the leak. Step intervention also tests whether the answer depends on the reasoning at all. The ablation that runs both on the same items settles which signal carries the detection.

Why not keep benchmarks private or verify answers with zero-knowledge proofs? P⁴Bench itself states that these proofs establish only that a valid answer was found, not how it was found. They also require problems that can be written as formal constraint systems. Neither approach audits scores already reported on public benchmarks.

Why not just call snap-back ordinary unfaithful reasoning? Clean models also skip steps, so the raw snap-back rate is confounded. That is why we use the model's own fresh-item rate as a baseline and validate in the clean-room setting. If the contaminated-versus-held-out gap vanishes after that correction, the hypothesis is refuted.

Decisive Checks

The idea succeeds if three things hold. In the clean-room setting, silent snap-back separates memorized from unseen items for models in the skip-step phase. In the main testbed, post-RL Stage II item-level AUROC clears 0.75 and beats ConStat under equal budget. And removing the consistency and repair dimensions measurably lowers AUROC.

The idea fails if either of two things happens. The snap-back gap may shrink under RL as badly as the loss gap does. Or snap-back rates on fresh items may already equal those on contaminated items, meaning the model's reasoning is unfaithful everywhere and snap-back says nothing about leakage.