Many ways of fine-tuning language models with reinforcement learning (RL) keep the model close to a trusted "reference" model. The usual tool is a penalty on the KL divergence, a measure of how far the new model's output distribution has moved from the reference's. The idea comes from expert-guided control, where it keeps agents human-like and safe. There is a problem when the reference model has memorized the test benchmarks. Then the penalty pays the new model to keep reproducing memorized answers. We propose a small change. On each training problem, the model is kept close to what the reference would say about a paraphrased version of that problem, which it does not recognize. It is not kept close to what the reference says about the original wording, which triggers recall. The anchor keeps its useful job of holding reasoning style and fluency in place. It stops rewarding recall of memorized answers.
This matters because contamination is not only an evaluation nuisance. Reasoning or Memorization? shows that RL with verifiable rewards (RLVR, where a program checks the final answer) raises Qwen2.5's MATH-500 scores even under random or inverted rewards. The reason is that training reactivates answers memorized during pre-training; the paper reports 65.8% exact match when the model is shown only 80% of the problem text. Spurious Rewards Paradox locates this recall in specific middle layers. Both papers diagnose the effect or repair it after training. Neither changes the training objective. Our claim is that the KL anchor itself is one of the channels that preserves the shortcut. A counterfactual anchor should remove the "free" benchmark gains while keeping real gains on clean problems.
Core Claim
Expert-guided regularization treats the reference as trustworthy everywhere. Examples include the human-anchored KL search in Diplodocus and KL-regularized search, the imitation anchor in RetroAct, and the action filters in the AI Clinician. The contamination literature shows the reference is untrustworthy in one specific way: its answers on inputs it recognizes. We therefore separate which input the reference is consulted on from which input the policy is trained on. Let $x$ be a training problem, $\tilde{x}$ a meaning-preserving paraphrase of it, $\pi$ the policy being trained, $\pi_{\text{ref}}$ the frozen reference, $r(x,y)$ the verifier reward for response $y$, and $\beta$ the penalty strength. We optimize
$$ J(\pi) = \mathbb{E}_{x}\,\mathbb{E}_{y\sim\pi(\cdot\mid x)}\big[r(x,y)\big] \;-\; \beta\,\mathbb{E}_{x}\,\mathrm{KL}\big(\pi(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid \tilde{x})\big). $$
A paraphrase keeps the correct answer the same, so both distributions live over the same set of responses and the KL is well defined. On problems the reference does not recognize, $\pi_{\text{ref}}(\cdot\mid\tilde{x})\approx\pi_{\text{ref}}(\cdot\mid x)$, and the objective reduces to standard KL-regularized RL. On recognized problems, the policy is pulled toward the reference's reasoning mode instead of its recall mode.
The test of this claim is concrete. Under random rewards, standard GRPO (group relative policy optimization, the common RLVR algorithm) on Qwen2.5 still gains on MATH-500. Under our anchor, that spurious gain should largely disappear. Gains under correct rewards on leakage-free sets should survive.
Research Program
The first component is building paraphrases that actually break recognition. We will rewrite each training and evaluation problem offline. The rewrites rename entities, reorder clauses and restate conditions, but keep every number and the answer. We keep a paraphrase only if it passes two checks. First, a strong solver must reach the same answer on both versions. Second, the reference's partial-prompt completion rate on the paraphrase must fall to near zero. We use that metric from Reasoning or Memorization? as our operational measure of "recognized." There is a known limit. On The Fragility of Benchmark Contamination Detection finds that contamination with full chain-of-thought traces teaches reasoning patterns that generalize to similar unseen questions. Paraphrasing may therefore fail to break semantic-level memorization. We will measure this directly rather than assume it away.
The second component is the training variant. It is GRPO in which only the input fed to the reference in the KL term changes. The cost is unchanged: the reference still gets one forward pass per sample. We also test a gated version, which uses $\tilde{x}$ only where the reference's partial-prompt completion on $x$ exceeds a threshold. The comparison tells us whether it matters to target the substitution.
The third component is analysis. We will check whether the policy's internal state on leaked items still passes through the recall route. We reuse the probe-gated audit and the layer-18 to 20 signature from Spurious Rewards Paradox. That paper removes the route after training; we ask whether the objective prevents it from being strengthened during training.
Baselines
The naive reference point is the untuned base model, Qwen2.5-Math-7B, plus the same pipeline run on a Llama model of similar size. Reasoning or Memorization? reports that Llama shows no spurious-reward gains, so it serves as a low-contamination control. On that control our method should behave exactly like standard GRPO. If it does not, the anchor is doing something other than removing recall.
The primary competitor is standard GRPO with the usual KL penalty to $\pi_{\text{ref}}(\cdot\mid x)$. This is the default expert-anchored objective, and the only difference from ours is the anchor's input. It will be run with identical data (DAPO-Math-17K prompts, the source pool used by MATH-Beyond), identical rollouts, steps and seeds, and the same sweep over $\beta$ for both methods. We found no reported result on Lacuna for standard GRPO accuracy on LiveMathBench or on paraphrased MATH-500. The comparison is therefore run in-house.
The strongest existing remedy for recall shortcuts is post-hoc layer reset. Spurious Rewards Paradox reports that resetting layers 18 to 20 lowers leaked MATH-500 accuracy from 98% to 86% while leaving LiveMathBench untouched. We apply that reset to the standard-GRPO checkpoint. Our method beats it if it closes more of the gap between canonical and paraphrased leaked accuracy, or the same amount, without any architecture-specific surgery and with equal clean accuracy.
The strongest existing regularizer for genuine exploration is adaptive entropy control. MATH-Beyond reports an expansion rate of 21.2% for Skywork-OR1-7B, against 5.22% for DeepScaleR-1.5B and 9.57% for Nemotron-v2. We reimplement adaptive entropy on our base and budget. Its published number comes from a different model and run, so it is not a direct comparison. We count as a win matching its expansion while also shrinking the leak gap.
The key ablation is paraphrase-only training. It uses standard KL with both policy and reference on $\tilde{x}$, which isolates whether the benefit comes from the anchor's counterfactual input or merely from exposure to paraphrases.
Why not just…?
Why not just drop the leaked problems from training? Filtering by n-gram overlap or partial-prompt audit removes known items from the RL corpus. It does not help, because the contamination is in the base model's pre-training, and training on other problems still activates recall on MATH-500. That is the core finding of Reasoning or Memorization?. It also misses semantic leakage, per Fragility. We will still run filtering as a baseline, and the random-reward check below will show whether it removes the spurious gain.
Why not just train on paraphrased prompts? This is a real threat. It is cheaper and could match us. Our prediction is that it will not, because it never shows the policy the original wording that triggers recall. Recall should therefore persist at test time on canonical benchmark text. Our objective trains on the original wording while anchoring to non-recall behavior. The paraphrase-only ablation settles this. If it closes the canonical leak gap as well as our method, the anchor adds nothing, and we will report that.
Why not just set β to zero or add an entropy bonus? Removing the anchor removes the channel we blame, but not the reward-driven reactivation. Spurious gains appear under standard GRPO regardless. Dropping the anchor also brings back the drift and instability that expert-anchoring exists to prevent; see the unstable "bang-bang" control that Reward-Constrained Behavior Cloning documents without it. The $\beta=0$ and entropy runs sit in the same sweep, so the data will show directly whether the anchor's input matters beyond its strength.
Why not just evaluate on dynamic benchmarks and ignore training? DyVal and RandomCalculation fix measurement, but only in formal domains. They do not stop a deployed model from using recall on real problems that resemble public ones. We use them as the clean yardstick, not as a replacement for fixing training.
Proposed Experiments
The decisive check is the random-reward test. We train Qwen2.5-Math-7B with random rewards under standard KL and under the counterfactual anchor, then measure MATH-500 accuracy change. Standard KL should reproduce the reported spurious gain. Ours should cut it substantially. On RandomCalculation, both should fail to improve under random reward, which confirms the benchmark is leakage-free.
The second check is the leak gap under correct rewards: canonical minus paraphrased accuracy on MATH-500, AMC and AIME 2024. This is paired with clean accuracy on LiveMathBench, RandomCalculation, OlymMATH and OmniMATH. We also report expansion rate on MATH-B-U (181 problems) and MATH-B-I (41 problems). Success means a smaller leak gap and clean accuracy within noise of standard GRPO or better.
The third check is the Llama control, where the method must be a no-op. We also report per-layer probe activity on leaked items, to see whether the recall route stops being strengthened.
A secondary experiment is the RL-MIA setting from Detecting Data Contamination from RL Post-training, where benchmark items are injected into the RL data. Here our anchor is not expected to stop direct learning of those items. We will say so, and report whether it changes the collapse signature that Self-Critique detects, which that paper reports at an average AUC of 0.70 on Qwen2.5-7B-Instruct.
The whole idea fails in one of two ways. Paraphrases might not break recognition, so the reference's partial-prompt completion on $\tilde{x}$ stays high. Or paraphrase-only training might match the method. Either outcome would be reported as the result.
References
- Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination
- Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
- Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning
- Modeling Strong and Human-Like Gameplay with KL-Regularized Search
- Improving Retrospective Language Agents via Joint Policy Gradient Optimization
- Understanding the Artificial Intelligence Clinician and optimal treatment strategies for sepsis in intensive care.
- On The Fragility of Benchmark Contamination Detection in Reasoning Models
- MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model
- Reward-Constrained Behavior Cloning
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models