When a benchmark question has leaked into a model's training data, the model may "know" the answer without working it out. Detecting this used to rely on the model being unusually confident on the leaked text. That signal has now largely disappeared for reasoning models. Reinforcement-learning post-training flattens the confidence gap between seen and unseen questions, and training on leaked chain-of-thought solutions (worked, step-by-step reasoning traces) teaches reasoning that carries over to near-duplicates. Under these conditions, ten leading detectors fall to near-chance accuracy.
We propose a behavioral detector that needs no access to the model's probabilities. We make a small edit to a benchmark question that changes its correct answer. Then we let the model alternate between two distinct roles: a solver that answers the edited question, and a critic that points to the first wrong step in the solver's reasoning. If the original answer is memorized, it should act as an attractor. Over rounds, the solver and critic should talk each other back to the old, now-wrong answer. For genuinely reasoned items, the loop should settle on the new answer.
The known weakness of mutual task refinement is that coupled tasks can reinforce each other's mistakes in an "echo chamber." Here that weakness becomes the measurement.
Core Claim
The claim is specific and falsifiable. On a benchmark item whose answer the model memorized, a bidirectional solver–critic loop run on answer-changing edits of that item returns to the original answer more often than a single solving pass does. The size of this amplification separates contaminated items from clean ones even in regimes where likelihood-based detectors are at chance.
Formally, let $x$ be a benchmark item with answer $a(x)$. Let $x'$ be an edit with a verified answer $a(x') \neq a(x)$. Let $\hat a_t(x')$ be the solver's answer after $t$ solver–critic rounds. The item's attractor score is
$$ S_T(x) = \Pr\big[\hat a_T(x') = a(x)\big] - \Pr\big[\hat a_0(x') = a(x)\big], $$
where the probability is over sampled edits and decoding seeds, and $T$ is the number of rounds. The second term is the single-pass "answer anchoring" rate. Subtracting it means the detector only earns credit for what the loop adds. That subtraction is the central design choice, because single-pass anchoring is the obvious simpler alternative.
Where this sits in prior work
The mutual-refinement literature supplies both the mechanism and the warning.
Heitz et al. show that feed-forward pipelines cannot correct an early error, because later stages never send information back (Cascaded Classification Models). Pose–action coupling (Turbo Learning Framework) and entity–relation recurrence (Recurrent Interaction Network) show that tentative outputs of one task can reshape another. Nie et al. find that one-directional adaptation can hurt the source task, so the feedback has to be reciprocal (Mutual Learning to Adapt).
All of these work on structured perception or extraction tasks and treat the confirmation loop as a failure to engineer away. We move the loop into language-model reasoning and treat that failure as a probe. We also borrow Nie et al.'s finding as a design constraint: our main configuration is bidirectional, and a one-way ablation tests whether the reciprocity matters here too.
On the contamination side, On The Fragility of Benchmark Contamination Detection in Reasoning Models motivates the work. It shows GRPO post-training and chain-of-thought contamination defeat likelihood detectors, and it calls for new detection paradigms without proposing one. Three other papers constrain the design:
- Min-K%++ (Min-K%++) normalizes token log-probabilities. That is exactly the signal RL clipping erases.
- CAP (CAP) measures consistency under meaning-preserving perturbations in one pass. Our edits instead change the answer, and our signal is the dynamics across rounds.
- ConStat (ConStat) compares performance against clean reference models. We need no reference models.
The closest work is MR-GSM8K. It shows models scoring above 80–90% on GSM8K score near zero when asked to grade and localize errors in given solutions. But it measures critique as a static, separate test on fixed candidate solutions from another model. We couple critique to the model's own solving of an edited item and read off the joint trajectory.
Self-Refine iterates self-feedback on a single task. Because it uses no distinct inverse task, it is our main loop ablation.
A Narrowing Geometry in Contaminated Reasoning finds that contaminated items route computation through a low-dimensional subspace. We use it as a mechanistic cross-check, not as a competitor.
Research Program
Evaluation substrate. The first component is a set of items whose answers can be edited with exact ground truth.
- MSCalc and DyVal graph-generated arithmetic, linear-equation and reachability tasks (DyVal) support fully programmatic edits. We change a leaf value or an edge and recompute the answer exactly.
- For GSM8K, we restrict to items where a reference program recomputes the answer after a numeric edit.
- For the competition sets used by the fragility paper (AIME, AMC, OlympiadBench, Minerva Math), we build a smaller human-verified subset of parameter edits, since automatic recomputation is not reliable there.
Each item gets several edits, including a "decoy-preserving" edit whose surface text stays close to the original.
The loop. In round $t$, the solver answers $x'$ with a worked solution. The critic receives $x'$ and that solution and must name the first incorrect step, or declare the solution correct, in the MR-GSM8K format. The solver then receives the critique and revises. The same weights play both roles through role prompts, so this detects contamination in a fixed model, not in a trained pair. We log each round's answer, the step the critic flags, and whether the critic's verdicts are consistent with the recomputed answer.
Beyond $S_T$, we record two secondary signals. The first is the round at which the trajectory first touches $a(x)$. The second is how often the critic endorses a solution that reaches $a(x)$ while mis-using the edited premise. That second signal is the echo-chamber signature.
Contamination with known labels. We follow the two-stage protocol of the fragility paper.
- Stage I: fine-tune a base model on a random half of the items (plus solutions), then run one epoch of GRPO on clean data.
- Stage II: contaminate DeepSeek-R1-Distill-Llama-8B directly with chain-of-thought traces.
Item-level membership is known, so we score detectors by AUROC (area under the curve of true- versus false-positive rate). We repeat this on MSCalc and DyVal items to get a clean, low-noise version of the same experiment.
Baselines
Naive reference. Random guessing gives 50% AUROC. It matters because the fragility paper reports that almost all ten detectors average "50-58% AUROC" in Stage II, so chance is the realistic floor.
Strongest existing detector. This is LiRA, as run in On The Fragility of Benchmark Contamination Detection in Reasoning Models. It catches supervised contamination at "89.1% AUROC" before RL, drops "by ~14%" after one GRPO epoch, and sits in the 50–58% band in Stage II. We will rerun LiRA and the loss detector with that paper's reference-model setup on our exact contaminated checkpoints and item splits. Beating it means exceeding LiRA's post-GRPO AUROC in Stage I and reaching at least 70% AUROC in Stage II on the same splits.
Reference-free likelihood detector. This is Min-K%++ (Min-K%++), run on the original item text with its published normalization. It is the strongest method that, like ours, needs no second model. No reported result was found on Lacuna for Min-K%++ on reasoning-model contamination specifically; its reported gain is "6.2% to 10.5%" AUROC over Min-K% on WikiMIA.
Perturbation- and reference-model detectors.
- CAP (CAP) is computed from single-pass accuracy on paraphrased, option-shuffled and distractor-altered versions. No reported AUROC was found on Lacuna for CAP on math reasoning benchmarks.
- ConStat (ConStat) uses clean reference models of similar size. It is reported to detect "98.4%" of sample-specific contamination. That is a benchmark-level statistic, so we compare it at the benchmark level only.
Ablations that isolate the key ingredient. All use the same edits, seeds and token budget as the full loop. We match compute by giving each ablation the same number of generated tokens.
- Single-pass anchoring rate $\Pr[\hat a_0(x') = a(x)]$ asks whether edits alone suffice.
- Majority vote over resampled solves asks whether extra samples alone suffice.
- Self-Refine, with the solver critiquing and revising its own answer, asks whether a distinct critique task matters.
- One-way loop, where critic feedback reaches the solver but the critic never sees the revision, asks whether reciprocity matters.
- Static solve-minus-critique gap, in MR-GSM8K style, is computed on the original items.
The proposal wins only if the full loop beats all five by a margin larger than the seed-to-seed spread.
Why not just…?
Why not just check whether the model gives the old answer to an edited question in one pass? This is the most serious threat. It may be enough on easy items, where memorization is so strong the model ignores the edit outright. The attractor score subtracts exactly this quantity, so if the loop adds nothing, $S_T$ collapses to zero and the ablation reports it. Our expectation of an advantage rests on two things:
- Chain-of-thought contamination makes models generalize to near-neighbors (Fragility), so a careful single pass will often read the edit correctly.
- The memorized answer should resurface only when the model is pushed to re-justify its steps.
If single-pass anchoring matches the loop's AUROC in Stage II, we will drop the loop and report edit-anchoring as the method.
Why not just use MR-GSM8K-style critique scores directly? A static solve-versus-critique gap measures the model's general weakness at grading, which MR-GSM8K shows is large even for clean models ("near zero" MR-Score for specialized math models). So it is confounded with ability, not with membership. Our item-level contrast compares contaminated and clean items within one model. The static-gap ablation tests directly whether coupling adds information beyond that confound.
Why not just use Self-Refine or more samples? Self-refinement on the same task inherits the model's blind spots (Self-Refine). It may converge to the memorized answer on clean items too, which would erase the contrast. Majority voting increases coverage but asks no question about the steps. Both run at matched compute, so the comparison settles this empirically.
Why not just suppress the memorization subspace? Top-Subspace Suppression needs weight access and an eigen-decomposition. It recovers genuine reasoning for "approximately half of the leaked samples" (Narrowing Geometry), and it is a mitigation, not a detector that works through an API. We use it as validation instead. If items with high $S_T$ are the ones whose answers change most under that suppression, the behavioral signal is tracking the same memorization structure.
Proposed Experiments and Decisive Checks
Main result. This is detection AUROC in Stage I after GRPO and in Stage II, across the competition subsets, GSM8K, MSCalc and DyVal, against every baseline above. Secondary metrics are true-positive rate at 5% false-positive rate, and the curve of AUROC against round count $T$. If attractors are real, AUROC should rise with $T$ and then plateau.
Contradiction test. The fragility paper implies some contaminated items are truly generalized. On those, the loop should not pull back, and arguably nothing needs detecting. We test this by correlating per-item $S_T$ with per-item inflation, meaning the contaminated model's accuracy minus the uncontaminated twin's accuracy on the original item. A detector that flags items without inflation is measuring something other than harmful leakage. We count that as a failure even if AUROC is high.
Controls.
- A clean-model control confirms that attractor scores on never-seen items stay near zero, including on the hardest items. Hardness alone could plausibly cause drift toward any salient number.
- A reciprocity check compares the bidirectional and one-way loops. This tells us whether Nie et al.'s finding that feedback must flow both ways carries over from perception tasks to reasoning.
- We report the added inference cost per item, since iterative loops are expensive. The known latency limitation of mutual refinement belongs in the result, not a footnote.
The idea is refuted if two conditions hold together: single-pass anchoring matches the loop, and the loop fails to beat 58% AUROC in Stage II.
References
- Cascaded Classification Models: Combining Models for Holistic Scene Understanding
- Turbo Learning Framework for Human-Object Interactions Recognition and Human Pose Estimation
- Recurrent Interaction Network for Jointly Extracting Entities and Classifying Relations
- Mutual Learning to Adapt for Joint Human Parsing and Pose Estimation
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- Min-K%++: Improved Baseline for Pre-Training Data Detection from Large Language Models
- CAP: Data Contamination Detection via Consistency Amplification
- ConStat: Performance-Based Contamination Detection in Large Language Models
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation
- Self-Refine: Iterative Refinement with Self-Feedback
- A Narrowing Geometry in Contaminated Reasoning
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks