Reinforcement learning with verifiable rewards (RLVR) trains a language model on problems whose answers a program can check, such as math with a symbolic checker or SQL with a database. The model is rewarded only when the checker says it is right. A known side effect is that the model narrows onto a few answer styles and forgets things it used to know. Recent fixes swap the usual penalty for a forward KL penalty. Forward KL is a "mass-covering" divergence: it punishes the model whenever it stops putting probability on outputs that a reference distribution, usually the original base model, still produces. These fixes measure success by Pass@k, the chance that at least one of k samples is correct, on static benchmarks like MATH500, AIME and BIRD.
That design has an unexamined flaw. A forward-KL penalty preserves whatever the reference model covers. The problems where the base model is already near-perfect, which are exactly the problems these methods select for rehearsal, are disproportionately likely to be ones it has memorized from leaked benchmark text. We propose to measure whether forward-KL regularization keeps memorized modes rather than reasoning modes, and to fix the anchor if it does. The fix replaces "base model on the original problem" with verified solutions on perturbed twins of each problem: symbol-remapped or premise-reordered versions that memorization cannot solve. For hard problems where the base model has no correct samples at all, a small role-decomposed agent team produces the anchor solutions, checked step by step against intermediate values from a programmatic problem generator.
Core Claim
The diversity that forward-KL RLVR preserves is only as clean as its anchor. The claim has two parts. First, measured on contamination-controlled twins, the Pass@k advantage of current forward-KL methods over reverse-KL training will shrink, because part of what they rehearse is memorized surface form. Second, building the anchor from twin-verified trajectories, plus team-decomposed solutions for problems the base model cannot solve, recovers a gain that survives perturbation. This also addresses the known "cold-start" failure of group-level forward-KL. Both halves can be falsified, and the first is worth knowing on its own.
Grounding in Prior Work
Two papers supply the mechanism we study.
- The Choice of Divergence introduces DPH-RL.
- It splits training data into problems the base policy nearly always solves and harder exploration problems.
- On the first set it applies forward-KL or Jensen–Shannon rehearsal against pre-sampled base trajectories; on the second it relaxes the penalty.
- It reports Pass@8 gains and less out-of-domain forgetting than reverse-KL GRPO (Group Relative Policy Optimization, the standard group-baseline policy-gradient method).
- Beyond Mode Collapse introduces DMPO.
- It approximates forward KL by fitting the policy to a Boltzmann reweighting of each sampled group, using a squared-error loss.
- It names the case where a group contains no correct samples as an open limitation.
Our proposal reuses their objectives unchanged and changes only where the reference mass comes from. Both papers evaluate on static sets and do not ask whether the rehearsed set is contaminated.
Three papers on contamination motivate the concern and supply the perturbations.
- A²RBench generates formally verified abstract tasks. It finds a 17.7% drop when familiar symbols are remapped.
- DyVal generates reasoning problems on the fly from directed acyclic graphs (DAGs).
- Models strong on static benchmarks, such as WizardMath-13B, score near zero there.
- Reversing premise order costs GPT-4 13.6% on reachability.
- Back to Square One shows similar reliance on scraped evaluation text on Winograd schemas.
All three use perturbation only for evaluation. We use it to decide what a training regularizer is allowed to preserve.
Multi-agent decomposition supplies the cold-start fix, and two findings constrain it.
-
MACM splits math solving into a Thinker that proposes conditions, a Judge that admits them, and an Executor that computes with code. It lifts GPT-4 Turbo on Level 5 MATH from 54.68% to 76.73%. - Self-collaboration Code Generation warns that models under roughly 13B cannot sustain distinct personas.
-
We therefore do not ask a small model to play Judge from its own judgment. On DyVal-style problems every DAG node has a known value, so the generator itself acts as the Judge, and the Executor is a code interpreter.
- Dr. MAS trains such teams directly with per-agent advantage normalization. We treat it as a competing way to use the team, not as our method.
Research Program
Twin construction
For every training problem $x$ we build perturbed twins $x'$ that keep the logical content but break surface familiarity:
- symbol remapping, as in A²RBench;
- premise reordering and reversal, as in DyVal;
- for SQL, consistent renaming of table and column identifiers with the database re-executed.
Each twin is checked by the same verifier as the original.
Memorization flag and twin anchor
We define a problem's memorization gap as the base model's success rate on $x$ minus its mean success rate on the twins.
- Problems with a large gap are flagged as likely memorized.
- For every problem, the anchor trajectories are sampled from the base model on the twins and kept only if verified. The rehearsed mass is therefore mass the base model can reproduce without the familiar string.
Team anchor for cold-start problems
For problems where neither the base model nor its twins yield any verified sample, we run a same-size Thinker/Executor team.
- On generated DAG problems, the generator verifies each intermediate node value.
- On math, a Judge pass checks executable sub-claims.
- Fully verified team trajectories, flattened into a single reasoning trace, enter the anchor.
Training objective
The resulting objective, with $\pi$ the policy, $J(\pi)$ the usual verifiable-reward RL objective, $\mathcal{A}(x)$ the verified anchor set for problem $x$, and $\beta$ the rehearsal weight, is
$$ \mathcal{L}(\pi) = -J(\pi) + \beta \, \mathbb{E}_{x}\, \mathbb{E}_{y \sim \mathcal{A}(x)}\left[-\log \pi(y \mid x)\right]. $$
The second term is a sample estimate of forward KL from the anchor distribution to the policy, up to a constant. The DPH-RL code base, seamoke/DPH-RL, already implements this with base-model samples. Our change is the contents of $\mathcal{A}$.
Primary metric
Our primary metric is twin-adjusted Pass@k: Pass@k computed only on twins never seen in training. We report it alongside the original-problem Pass@k and their difference.
Baselines
Base model (do nothing). The untrained base model is the reference for both raw accuracy and the memorization gap. Every method must beat it on twin-adjusted Pass@k to count as learning anything.
Reverse-KL GRPO. This is the standard reverse-KL GRPO recipe. It is the reference point against which DPH-RL and DMPO report their gains.
DPH-RL, the strongest existing method. The Choice of Divergence is the strongest competitor because it already uses forward KL against a base-model anchor on the same benchmarks we target. It reports Pass@8 on BIRD "improved by 4.3% over the GRPO baseline." We run it from the released code with its own data partition, on identical training problems, rollout counts and GPU hours. Both methods get the same hyperparameter sweep over $\beta$.
DMPO. We include DMPO from Beyond Mode Collapse on the combinatorial and math settings. It reports 43.9% Quality Ratio on NP-Bench against GRPO's 40.1%.
Beating the baselines. "Beating" DPH-RL means a higher twin-adjusted Pass@8 at matched compute, with original-problem Pass@1 no lower than DPH-RL's minus a pre-registered tolerance of one point.
Ablations. Three ablations isolate the key ingredients:
- Twin-only: the twin anchor without team trajectories. It tests whether cold-start coverage matters.
- Original-anchor with team: team trajectories added to an anchor built on unperturbed problems. It tests whether the twin filter matters.
- Best-of-N single model: the team's cold-start contribution replaced by a single model sampling at matched token budget. It tests whether decomposition, rather than extra compute, finds the solutions.
Why not just…?
Why not just add an entropy bonus or raise sampling temperature? This addresses diversity but not which modes survive. A random token-level spread would keep memorized and reasoned solutions alike. Beyond Mode Collapse reports that its forward-KL method ends at lower entropy than GRPO while producing more diverse correct paths, which suggests unstructured entropy is not the right lever. We still run an entropy-bonus GRPO arm at matched tuning budget and report its twin-adjusted Pass@k.
Why not just decontaminate the training data with n-gram filters? Filtering removes known overlaps from our training data. It does nothing about what the base model already memorized during pretraining, and that pretrained knowledge is precisely what forward-KL rehearsal anchors to. The A²RBench remapping drop and DyVal's near-zero scores for statically strong models indicate that contamination lives in the model, not just the dataset. An n-gram-filtered DPH-RL arm tests this directly. If it closes the twin gap as well as our anchor, the twin construction is unnecessary, and we will report that.
Why not just fine-tune on the team's verified traces and then run GRPO? This is the most serious threat. Supervised training on samples from a distribution is forward-KL minimization toward it, so a two-stage pipeline could capture most of the benefit. The difference is timing:
- In the two-stage pipeline, the subsequent reverse-KL phase is free to collapse the coverage it just learned.
- In our method, the anchor stays in the loss throughout.
We run supervised-then-GRPO and rejection-sampling fine-tuning with identical anchor data and compute. If either matches our method on twin-adjusted Pass@k, the contribution reduces to "use a twin-filtered, team-augmented dataset." That would still be a useful finding, and we would state it that way.
Why not just train the team directly with Dr. MAS and deploy the team? Dr. MAS reports +5.6% avg@16 on math for trained multi-agent systems. However, a deployed team multiplies inference cost, a cost MACM itself notes, and Dr. MAS was evaluated only on static sets. We run Dr. MAS on the same problems and report both twin-adjusted accuracy and tokens per solved problem. A team that wins on twins at acceptable cost would argue for deploying teams instead of distilling them.
Proposed Experiments
Settings.
- Math: train on Openr1-Math-46K-style problems; evaluate on MATH500 and AIME with remapped and reordered twins.
- SQL: BIRD and Spider with identifier-renamed twins.
- Procedural: freshly generated DyVal arithmetic, logic and reachability DAGs across difficulty tiers, plus Countdown, Zebra and ARC-1D generated with new seeds. These supply cold-start cases with verifiable intermediate values.
- Out-of-distribution: A²RBench, including its remapped split.
- Models: 7B-class open models, chosen so the cold-start regime is common.
- Sweep: three seeds per arm.
First decisive check (diagnostic). We test whether the problems DPH-RL rehearses carry a larger base-model memorization gap than the exploration problems, and whether DPH-RL's Pass@8 advantage over GRPO shrinks on twins.
- If its advantage on twins equals its advantage on originals, the contamination hypothesis is wrong for these benchmarks. We would then report the anchor as already clean and drop the twin filter.
Second decisive check (intervention). We test whether our objective beats DPH-RL on twin-adjusted Pass@8 at matched compute without losing original-problem Pass@1, and whether it beats supervised-then-GRPO on the same anchor data.
Third decisive check (cold start). On DyVal problems at the hardest tier where base groups contain no correct samples, we compare three things: the fraction of training groups with at least one verified sample, and the final accuracy, under the team anchor versus the best-of-N anchor. This distinguishes the value of role decomposition from that of extra sampling.
References
- The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
- Beyond Mode Collapse: Distribution Matching for Diverse Reasoning
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- MACM: Utilizing a Multi-Agent System for Condition Mining in Solving Complex Mathematical Problems
- Self-Collaboration Code Generation via ChatGPT
- Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems