On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation

Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun WeiarXiv 2026 · arXiv

TL;DR

While on-policy distillation (OPD) is highly effective at mimicking "expert" teachers on specific tasks, it often fails to transfer their broader generalization capabilities. This paper identifies a critical "blind spot" where near-perfect in-distribution performance masks significant out-of-distribution (OOD) forgetting, and proposes geometric patching and replay-based strategies to bridge this gap.

What Problem Does This Paper Address?

In the final stages of training large language models (LLMs), researchers often use capability consolidation to merge specialized "expert" models (e.g., a math expert and a coding expert) into a single, versatile student model. On-Policy Distillation (OPD) is a go-to method for this, where a student model learns from its own generated trajectories under the supervision of a teacher.

The problem is that "success" is usually measured by how well the student matches the teacher on the specific training data (In-Distribution or IND). The authors discovered that even when a student achieves near-perfect IND scores, it often loses the base model's original capabilities or fails to inherit the teacher's ability to solve harder, unseen problems (Out-of-Distribution or OOD). This is particularly problematic for extrapolative generalization (solving harder problems in the same domain) and compositional generalization (combining two skills, like coding within a long-context window).

Methods

The researchers conducted a systematic study using Qwen3-8B as the base student model. They compared several training pipelines:

  • Single-task & Multi-teacher OPD (MOPD): Distilling from one or more specialized teachers using a reverse-KL loss on student-generated tokens.
  • Mixed RL: A baseline where the model is trained directly on multiple tasks using Reinforcement Learning (specifically GRPO) with verifiable rewards.
  • Geometric Analysis: They analyzed the "geometry" of parameter updates ($\Delta W$), looking at the norm (size) and spectral concentration (how many directions the update affects) of the changes made to the model's weights.
  • Mitigation Strategies:
    • Procrustes-attention patching: A post-hoc method that aligns the student's updates with the teacher's updates to prune "noise" that harms OOD performance.
  • Base-preserving replay: Mixing the teacher's training data with examples where the student is supervised by its own original "base" version to prevent forgetting.

Datasets

The study spans several diverse domains:

  • Mathematical Reasoning: Training on GSM8K and MATH; evaluating on harder benchmarks like AMC23 and AIME24/25.
  • Coding: Training on Nemotron competitive-coding data; evaluating on HumanEval and MBPP.
  • Agentic Capabilities: Training on BFCL and ToolACE; evaluating on BFCL V4 live/non-live subsets.
  • Long Context: Training on DocQA; evaluating on LongBench-v2.
  • Synthetic Reasoning: A controlled study using "GSM-Infinite" to precisely measure how performance drops as problem complexity (operation count) increases.

Key Insights and Ideas

  • The IND Mirage: Near-perfect distillation on training tasks is a "blind spot." It can mask a massive failure to generalize. As shown in the synthetic study, as soon as the problem difficulty exceeds the training range, the student's performance relative to the teacher plummets.
Figure 1: The gap between IND and OOD performance in a controlled synthetic study.
  • The "Flask" Bottleneck: The authors use a flask analogy to explain OOD gaps. Generalization is constrained by two necks: Teacher Capability (the teacher must actually know the OOD skill) and Data Coverage (the training data must be broad enough for the student to "see" the connection). Improving only one is often insufficient.
  • OPD is "Compact": Interestingly, OPD produces much smaller and more "spectrally concentrated" updates than standard RL. While this helps MOPD avoid some of the "task interference" seen in mixed RL, it also means the model might not be "re-wiring" itself enough to truly inherit the teacher's underlying reasoning logic.
  • Compositional Failure: Improving on Task A and Task B separately does not guarantee the model can do Task A+B. For example, a model might get better at coding and better at long-context tasks, but still fail at "long-context coding."

Key Results

  • The OOD Gap: Across extrapolative math tasks, OPD and MOPD reached over 96% of their teachers' IND scores but remained 2.45–16.25% below the base model on OOD evaluations.
  • Compositional Gaps: In multi-teacher settings, MOPD's performance on composed tasks remained 18.19–24.34% below the best teachers, despite improving on the individual component tasks.
Figure 6: MOPD improves individual math and long-context scores but fails to reach teacher levels on the composed "Long-context Math" task.
  • Successful Mitigations:
    • Patching raised AIME25 scores by 4.17 percentage points while keeping 93.5% of the teacher's math accuracy.
  • Base-preserving replay improved OOD scores on 5 out of 8 benchmarks, even surpassing both the base model and the RL teacher on the difficult AIME25 benchmark.

Limitations and Future Work

The authors acknowledge that their study primarily focused on the Qwen3-8B model family. While preliminary tests on a much larger 235B model showed similar trends, the scaling laws of this "generalization gap" deserve more study.

Furthermore, the "OOD" labels used are relative to the post-training data, not necessarily the model's original pre-training exposure. Future work should explore more sophisticated "step-level" parameter controls during training—rather than just post-hoc patching—to explicitly steer models toward better generalization from the start.

Summary updated

Abstract

Multi-teacher on-policy distillation (MOPD) extends on-policy distillation (OPD) to consolidate capabilities from multiple teachers. Although prior work reports near-perfect in-distribution (IND) distillation performance, we find that this can mask incomplete capability consolidation beyond the supervised distribution. A preliminary study on synthetic reasoning shows that OPD does not fully transfer teachers' generalization capabilities. We then extend this investigation to real-world tasks, evaluating extrapolative and compositional generalization across mathematical reasoning, coding, agent, and long-context capabilities using Qwen3 models. Across extrapolative settings, OPD and MOPD reach 96.04-102.73% of their teachers' IND scores. Yet their out-of-distribution (OOD) scores remain 2.45-16.25% below the base model's. Across three compositional settings, MOPD improves IND performance over the base model, but its performance on composed tasks as OOD remains 18.19-24.34% below that of the best teachers. To understand these gaps, we examine teacher capability and data coverage as two potential bottlenecks. Jointly alleviating these bottlenecks eliminates OOD forgetting in our experiments and suggests the potential for generalization beyond the teacher. (M)OPD also reduces OOD forgetting on average relative to its reinforcement learning (RL) baselines. Analysis of the associated update geometry reveals smaller and more spectrally concentrated updates on average than those of the RL baselines, along with greater shared update structure under MOPD. Motivated by these findings, we propose geometry-based patching and base-preserving replay to improve OOD performance while largely preserving IND performance. Our findings identify a blind spot in capability consolidation via (M)OPD, highlighting the need for algorithms that explicitly support OOD retention and generalization.
BibTeX
@misc{wang2026generalization,
  author = {Yichen Wang and Yufa Zhou and Siqi Zhu and Haozheng Luo and Jingzhi Bao and Zhichao Liu and Zijun Wei},
  title = {{On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation}},
  year = {2026},
  eprint = {2611.05609},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05609},
}
Figure