CoRD: Coupled Rollouts Fill the Blind Spot of On-Policy Distillation

Guhan Chen, Bohan Li, Jun Rao, Hejin Wang, Songtao Tian, Yexin Xie, Xiaojun Meng, Peng Yuan, Jiansheng Wei, Xuebo Liu, Zixiong YuarXiv 2026 · arXiv

TL;DR

CoRD (Coupled-Rollout Distillation) addresses a fundamental "blind spot" in on-policy distillation where student models never learn from high-quality reasoning paths they are too weak to generate themselves. By using a "coupled sampler" borrowed from speculative decoding, CoRD steers student rollouts toward teacher-preferred states, leading to significantly better reasoning performance on math benchmarks without increasing the training budget.

What Problem Does This Paper Address?

In the world of Large Language Model (LLM) distillation, On-Policy Distillation (OPD) is a popular technique. The idea is simple: the student model generates its own "rollouts" (reasoning paths), and a stronger teacher model provides feedback by showing the student what the "correct" next-token distribution should have been at every step.

However, OPD has a major flaw: the blind spot. Because the student only learns from states it actually visits, it rarely receives supervision on high-quality reasoning paths that the teacher would take but the student considers unlikely. If a student is bad at math, it will generate "garbage" paths and only get feedback on how to make that garbage slightly better. It never sees the "golden" path to the solution. The authors found that before training, students are 1.5 to 2.4 times further from the teacher on teacher-preferred states than on their own generated states, and standard OPD is painfully slow at closing this gap.

Figure 1: Overview of CORD. (i) Left (OPD’s blind spot): On-policy distillation trains only on student rollouts. Teacher-preferred reasoning paths that the student rarely or never visits therefore receive little or no supervision. (ii) Right (our coupled sampler): CORD keeps the same training loss and budget, but changes the rollout sampler. By coupling the student with a teacher-leaning target qα, it preserves agreeing tokens and replaces disagreements with underweighted target tokens. This maximal coupling steers rollouts into teacher-preferred states missed by OPD with the minimum possible intervention rate, minimally perturbing the student’s original trajectory.

Methods

The authors introduce CoRD, which changes the rollout sampler while keeping the training loss and budget identical to OPD.

The core innovation is the Coupled Sampler. Instead of sampling purely from the student ($p_S$), CoRD samples from a "teacher-leaning target" ($q_\alpha$), which is a geometric interpolation between the student and teacher. To do this efficiently and minimally, they use the rejection sampling rule from speculative decoding:

  1. The student proposes a token.
  2. The sampler accepts the token with a probability based on how much the teacher likes it relative to the student.
  3. If rejected, it resamples a token from the "residual" mass—specifically the tokens the teacher overweights compared to the student.

This approach is a "maximal coupling," meaning it intervenes as little as possible to achieve the target distribution. Interventions only happen where the models disagree, and they naturally decrease as the student learns and aligns with the teacher.

Figure 2: One-step coupled sampling. On a three- token vocabulary with α = 0.6, student proposals are kept whenever possible and replaced only when neces- sary, so the emitted token follows qα exactly.

Datasets

The models were trained on the MATH training set (approx. 5,000 problems). Evaluation was performed across five public math benchmarks to test reasoning and generalization:

  • MATH500
  • GSM8K
  • AIME 2024 & 2025
  • AMC 2023

The authors tested six student models from the Qwen3 and Qwen3.5 families, ranging from 0.6B to 4B parameters, distilled from 8B and 9B teachers respectively.

Key Insights and Ideas

  • Steering, Not Just Perturbing: Simply adding noise or random interventions to the student's rollout doesn't help. The authors tested "DAgger-style" random teacher interventions and found they performed significantly worse than CoRD. The direction of the intervention—steering toward the teacher's specific preferences—is what drives the gain.
  • Closing the Gap Faster: CoRD is remarkably efficient. For a 4B student, CoRD reached a level of "gap closure" on teacher-reached states in just 16 steps that standard OPD couldn't reach even after 80 steps.
  • Automatic Scheduling: Because the intervention rate is tied to the Total Variation (TV) distance between the student and the target, the sampler "withdraws" automatically as the student improves. No manual hyperparameter tuning for an intervention schedule is required.
  • Speculative Decoding as a Training Tool: While speculative decoding is usually used to speed up inference, this paper cleverly repurposes its mathematical properties to solve a distribution mismatch problem in training.

Key Results

CoRD consistently outperformed standard on-policy distillation across all tested models and benchmarks:

  • Accuracy Gains: Average pass@1 scores increased by 0.9 to 2.0 percentage points, and pass@8 scores rose by 1.2 to 3.4 points.
  • Efficiency: CoRD outperformed OPD even when OPD was given twice the training steps.
  • Difficulty Breakdown: The largest gains were seen on problems the untrained student could rarely or never solve, proving that CoRD successfully helps students explore and learn from difficult reasoning paths they previously ignored.
Learning curves showing that CoRD maintains higher visited-state divergence (meaning it's still finding things to learn) and achieves higher held-out accuracy.

Limitations and Future Work

  • Compute Requirements: CoRD requires the teacher model's logits during the rollout phase of training. While the authors' implementation made this cost-neutral compared to their OPD baseline, it is more intensive than "offline" distillation methods that pre-generate data.
  • Tokenizer Constraints: The student and teacher must share the same tokenizer for the coupling logic to work.
  • Scope: The study focused exclusively on mathematical reasoning. While the results are promising, it remains to be seen how well CoRD generalizes to other domains like code generation or creative writing.
  • Model Scale: The experiments topped out at 4B parameter students and 14B teachers. Testing the scaling laws of CoRD on much larger models (e.g., 70B+) is a natural next step.

Summary updated

Abstract

On-policy distillation trains a student on its own rollouts, so states the teacher would reach but the student does not are seldom supervised. We introduce CoRD (coupled-rollout distillation), which keeps the on-policy loss and moves only the distribution of training states toward the teacher, drawing rollouts from a geometric interpolation of student and teacher through the maximal coupling of speculative decoding. The coupling keeps the student's token wherever the two distributions agree and otherwise substitutes a token from the part of the target the student underweights, so interventions are as rare as any exact sampler allows and recede as the student learns. Across six base students from two model families (Qwen3 and Qwen3.5, 0.6B to 4B), CoRD raises average pass@1 over on-policy distillation by 0.9 to 2.0 points and pass@8 by 1.2 to 3.4 points on five public math benchmarks at equal steps and data. On frozen teacher-reached states, a 4B student trained with CoRD reaches in 16 steps a gap closure that on-policy distillation does not reach in 80, and substitution rules intervening at least as often in other directions fall short of CoRD.
BibTeX
@misc{chen2026cord,
  author = {Guhan Chen and Bohan Li and Jun Rao and Hejin Wang and Songtao Tian and Yexin Xie and Xiaojun Meng and Peng Yuan and Jiansheng Wei and Xuebo Liu and Zixiong Yu},
  title = {{CoRD: Coupled Rollouts Fill the Blind Spot of On-Policy Distillation}},
  year = {2026},
  eprint = {2611.07959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07959},
}
Figure