Twin Rollouts: Contamination Detection Inside GRPO's Sampling Budget

Reinforcement learning with checkable rewards now drives much of the progress in math and code reasoning. Its most common algorithm, Group Relative Policy Optimization (GRPO), samples a group of answers per problem and scores each answer against the group average. This training step also hides benchmark leakage. After GRPO, the likelihood signals that contamination detectors rely on fade away, while the inflated scores remain. Detectors that do survive this, such as asking the model for an alternative solution or re-running a perturbed copy of the benchmark, each require extra full inference passes. We propose a cheaper test. Take the $G$ samples that GRPO (or pass@k evaluation) already draws for each problem. Spend half of them on the original problem and half on a "twin": a version whose numbers or inputs have been changed so the correct answer changes, with the new answer recomputed by a small verified program. A model that reasons should score about the same on both halves. A model that retrieves a memorized answer will do well on the original and poorly on the twin. The gap between the two halves is a per-problem memorization score, and it adds no rollouts beyond the ones already paid for.

This combines three strands of work. Dynamic evaluation supplies the twins. GRPO's group sampling supplies a budget that can pay for them. Contamination research explains why a behavioral signal is needed at all. The main engineering question is the overhead that has kept dynamic evaluation out of training loops, and our answer is to split the cost into two parts. An expensive step runs once per problem: writing and verifying a parametrized solver program. A cheap step runs for every twin: executing that program with new parameters.

Core Claim

A twin gap computed inside a fixed sample budget will detect contamination that GRPO has hidden. It should do at least as well as Self-Critique while costing no extra tokens relative to standard $G$-sample rollouts. The twins must change the answer; twins that keep the answer fixed will fail. For problem $q$, let $\bar r_{\text{orig}}(q)$ be the mean verified reward over the $G/2$ rollouts on the original, and $\bar r_{\text{twin}}(q)$ the mean over $G/2$ rollouts spread across freshly sampled twins. The score is

$$g(q) = \bar r_{\text{orig}}(q) - \bar r_{\text{twin}}(q),$$

and large positive $g$ flags memorization. Part of the signal comes from reward and part from the model's uncertainty, so we also record the token entropy of each half.

We expect this to survive the loss of the likelihood gap for a specific reason. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that GRPO's clipping term closes the log-probability gap between seen and unseen items. That argument concerns how likely the model finds the text. Our score measures whether answers are correct on counterfactual inputs, which that argument does not reach.

Where This Sits

Three papers motivate the gap. Fragility shows that contamination introduced during supervised fine-tuning is hidden by a clean GRPO phase: one epoch cuts LiRA AUROC by about 14%, and contamination with chain-of-thought solutions leaves all ten detectors it tests at 50–58% AUROC. That paper offers no working detector.

Reasoning or Memorization? shows that GRPO with random rewards raises Qwen2.5's MATH-500 score by retrieving memorized answers. Its evidence is 65.8% exact match from only an 80% prompt prefix, and the same gains collapse on its freshly generated RandomCalculation problems. That comparison between an original set and a generated set is the benchmark-level version of our per-problem twin gap.

Detecting Data Contamination from RL Post-training introduces Self-Critique, which detects the rigid, low-entropy behavior RL produces on memorized items. It costs two full generations per item.

The method itself comes from dynamic evaluation. DyVal generates problems from graphs with controlled structure. Is Your Benchmark Still Useful? mutates code while preserving its meaning and records drops of up to 40% on CRUXEval. CAP compares consistency between original and perturbed benchmarks.

We depart from all three in two ways. First, their perturbations mostly keep the answer fixed, and a model that has memorized the answer can pass them. Second, they run as a separate offline pass, so they double the evaluation cost. Twin rollouts change the answer and live inside the existing sample budget. Your Group-Relative Advantage Is Biased limits the design: statistics from small groups are noisy, so the detection-power analysis below has to account for $G/2$-sized halves.

Research Program

Building the twins. We work only in domains where a changed answer can be recomputed exactly.

  • Code. CRUXEval-O (CRUXEval): we resample function inputs and re-execute the program.
  • Logic. Knights & Knaves and SAT instances, regenerated from their generators.
  • Arithmetic. RandomCalculation-style expressions.
  • Competition math. A subset of MATH-500 and AIME. For each problem, an LLM writes a parametrized Python solver once. We keep the solver only if it reproduces the gold answer on the original parameters and gives stable outputs when parameters are perturbed slightly. We also keep only parameter ranges that give well-posed problems.

We report the one-time cost and the fraction of problems that survive this filter. A low survival rate on competition math would be an honest limit on the method's reach.

Using twins in evaluation. Standard pass@k evaluation samples $k$ answers on the original. We keep the same $k$ and split it $k/2$ original, $k/2$ twin. Compared with pass@k the extra cost is zero. Compared with greedy decoding it is not zero, and we say so.

Using twins in training. Each group is split the same way. For the policy update, advantages are normalized separately within each half, so that failing twins do not inflate the advantage of memorized successes on the original. The twin gap $g(q)$ is logged at every step as a live memorization monitor.

We also test one intervention, a coupled reward. A rollout on the original counts as correct only if the twin half of the same group reaches a success threshold. The intent is to stop GRPO from reinforcing answer retrieval.

A side effect is that twin halves restore group variance. On a memorized problem where every original rollout succeeds, the group has zero spread and standard GRPO gets no gradient. We measure how often this happens.

Baselines

Likelihood detectors (naive reference). Min-K% Prob, loss thresholding and LiRA, run on the same checkpoints. They cost one forward pass. The reported points are 0.42–0.53 AUC for PPL and Min-K% on RL-contaminated models (Detecting Data Contamination from RL Post-training) and near-random AUROC after contamination with chain-of-thought data (Fragility). Clearly exceeding these is necessary but not enough.

Self-Critique (strongest existing method). This is the strongest competitor because it was built for post-RL contamination and is also behavioral. It reports average AUC 0.70 on Qwen2.5-7B-Instruct and 0.64 on DeepSeek-Math-7B-Instruct on RL-MIA. We will run it with its own two-generation protocol, and again at a token budget matched to ours, for example the same total generated tokens spread over repeated critiques. Beating it means higher AUROC at matched tokens on RL-MIA and on our Stage I replication. Matching its AUROC at lower cost counts as a partial win.

CAP. Run with option shuffling and paraphrase perturbations on the same items and the same sample budget. No reported result was found on Lacuna for CAP on GRPO-trained reasoning models.

Key-ingredient ablation. Twins that keep the answer fixed: variable renaming, paraphrase, constant unfolding. These are otherwise identical to our pipeline. If they match answer-changing twins, our central mechanistic claim is wrong.

Training baselines. The training comparisons use the same GRPO hyperparameters, prompts and compute:

  • standard GRPO on the original prompts;
  • GRPO trained only on twins;
  • our split-group GRPO, with and without the coupled reward.

Why not just…?

Why not just scrub the training data with n-gram filters? The evidence says this is incomplete. CAP found 21 leaked questions in Baichuan-13B that had passed the filters. Fragility shows that chain-of-thought contamination generalizes beyond the surface strings. Filtering also says nothing about leakage in pre-training data that the RL practitioner never sees, which is exactly the Qwen2.5 case.

Why not just evaluate on fresh benchmarks like LiveMathBench or AIME 2025? Fresh sets are useful, and we use LiveMathBench as an outcome measure. But they give a benchmark-level verdict with no per-item attribution. They are also small, and they leak once they are published. MATH-Beyond shows that base models solve nearly all of AIME24 at pass@1024, so fixed sets lose resolving power quickly. Twins give an unlimited supply of new instances per item.

Why not just run Self-Critique or CAP offline after training? This is the main threat on detection. If Self-Critique matches the twin gap at equal tokens, our advantage reduces to reusing samples during training, which is a smaller claim. The matched-budget comparison settles it. We also expect the two signals to fail in different ways: Self-Critique looks for rigid behavior, while twins test counterfactual correctness. So we report a combined detector as well.

Why not just train GRPO only on twins and skip the diagnosis? This is a real threat to the training contribution, and it may remove spurious gains just as well. If it does, we report that. The coupled reward would then be unnecessary, and the contribution becomes the zero-cost monitor plus the finding that twins suffice. What training only on twins cannot do is tell you which training problems your model had memorized. It also discards the original distribution, which may hurt performance on natural-language phrasings. We measure that cost on LiveMathBench.

Proposed Experiments

Detection under hiding by GRPO. We replicate Fragility's Stage I:

  1. Fine-tune Qwen2.5-7B-Instruct on half of each twin-capable set, so that half is contaminated.
  2. Run clean GRPO for one to three epochs.
  3. After each epoch, score AUROC of $g(q)$ at $k \in \{4, 8, 16\}$ against all the baselines above.

We repeat this in the RL-MIA setting, where 50% of test items are injected into GRPO training. The decisive outcome is simple. Likelihood-detector AUROC should decay across epochs, as Fragility reports, while twin-gap AUROC stays flat. If twin-gap AUROC also decays toward 0.5, then GRPO hides behavioral signals too, and the idea fails.

Natural contamination. On the programmable subset of MATH-500, we compute $g(q)$ for Qwen2.5-Math-7B and a Llama model. The twin gap should be larger for Qwen2.5. Per item, it should correlate with partial-prompt answer accuracy from Reasoning or Memorization?, which gives an external check without injected labels.

Training under spurious rewards. We run GRPO on MATH training problems with correct, random and inverted rewards. We track $g(q)$ during training. The prediction is that it rises under random rewards on Qwen2.5 and stays flat for Llama. We then compare the three training variants on RandomCalculation, LiveMathBench and MATH-B.

On MATH-B, the reported expansion rates for RLVR models are 5.22% (DeepScaleR-1.5B) and 9.57% (Nemotron-v2). We do not expect twins to raise expansion much. A null result there would support the view that twins remove memorization but do not create new capability.

Overhead accounting. For each method we report:

  • total generated tokens;
  • wall-clock time per training step;
  • the one-time LLM cost of writing solvers, amortized over the number of twins drawn.

The claim is supported only if the per-step overhead of twin execution is small relative to generating the rollouts themselves.