TL;DR
While on-policy distillation (OPD) is highly effective at mimicking "expert" teachers on specific tasks, it often fails to transfer their broader generalization capabilities. This paper identifies a critical "blind spot" where near-perfect in-distribution performance masks significant out-of-distribution (OOD) forgetting, and proposes geometric patching and replay-based strategies to bridge this gap.
What Problem Does This Paper Address?
In the final stages of training large language models (LLMs), researchers often use capability consolidation to merge specialized "expert" models (e.g., a math expert and a coding expert) into a single, versatile student model. On-Policy Distillation (OPD) is a go-to method for this, where a student model learns from its own generated trajectories under the supervision of a teacher.
The problem is that "success" is usually measured by how well the student matches the teacher on the specific training data (In-Distribution or IND). The authors discovered that even when a student achieves near-perfect IND scores, it often loses the base model's original capabilities or fails to inherit the teacher's ability to solve harder, unseen problems (Out-of-Distribution or OOD). This is particularly problematic for extrapolative generalization (solving harder problems in the same domain) and compositional generalization (combining two skills, like coding within a long-context window).
Methods
The researchers conducted a systematic study using Qwen3-8B as the base student model. They compared several training pipelines:
- Single-task & Multi-teacher OPD (MOPD): Distilling from one or more specialized teachers using a reverse-KL loss on student-generated tokens.
- Mixed RL: A baseline where the model is trained directly on multiple tasks using Reinforcement Learning (specifically GRPO) with verifiable rewards.
- Geometric Analysis: They analyzed the "geometry" of parameter updates ($\Delta W$), looking at the norm (size) and spectral concentration (how many directions the update affects) of the changes made to the model's weights.
- Mitigation Strategies:
- Procrustes-attention patching: A post-hoc method that aligns the student's updates with the teacher's updates to prune "noise" that harms OOD performance.
- Base-preserving replay: Mixing the teacher's training data with examples where the student is supervised by its own original "base" version to prevent forgetting.
Datasets
The study spans several diverse domains:
- Mathematical Reasoning: Training on GSM8K and MATH; evaluating on harder benchmarks like AMC23 and AIME24/25.
- Coding: Training on Nemotron competitive-coding data; evaluating on HumanEval and MBPP.
- Agentic Capabilities: Training on BFCL and ToolACE; evaluating on BFCL V4 live/non-live subsets.
- Long Context: Training on DocQA; evaluating on LongBench-v2.
- Synthetic Reasoning: A controlled study using "GSM-Infinite" to precisely measure how performance drops as problem complexity (operation count) increases.
Key Insights and Ideas
- The IND Mirage: Near-perfect distillation on training tasks is a "blind spot." It can mask a massive failure to generalize. As shown in the synthetic study, as soon as the problem difficulty exceeds the training range, the student's performance relative to the teacher plummets.
- The "Flask" Bottleneck: The authors use a flask analogy to explain OOD gaps. Generalization is constrained by two necks: Teacher Capability (the teacher must actually know the OOD skill) and Data Coverage (the training data must be broad enough for the student to "see" the connection). Improving only one is often insufficient.
- OPD is "Compact": Interestingly, OPD produces much smaller and more "spectrally concentrated" updates than standard RL. While this helps MOPD avoid some of the "task interference" seen in mixed RL, it also means the model might not be "re-wiring" itself enough to truly inherit the teacher's underlying reasoning logic.
- Compositional Failure: Improving on Task A and Task B separately does not guarantee the model can do Task A+B. For example, a model might get better at coding and better at long-context tasks, but still fail at "long-context coding."
Key Results
- The OOD Gap: Across extrapolative math tasks, OPD and MOPD reached over 96% of their teachers' IND scores but remained 2.45–16.25% below the base model on OOD evaluations.
- Compositional Gaps: In multi-teacher settings, MOPD's performance on composed tasks remained 18.19–24.34% below the best teachers, despite improving on the individual component tasks.
- Successful Mitigations:
- Patching raised AIME25 scores by 4.17 percentage points while keeping 93.5% of the teacher's math accuracy.
- Base-preserving replay improved OOD scores on 5 out of 8 benchmarks, even surpassing both the base model and the RL teacher on the difficult AIME25 benchmark.
Limitations and Future Work
The authors acknowledge that their study primarily focused on the Qwen3-8B model family. While preliminary tests on a much larger 235B model showed similar trends, the scaling laws of this "generalization gap" deserve more study.
Furthermore, the "OOD" labels used are relative to the post-training data, not necessarily the model's original pre-training exposure. Future work should explore more sophisticated "step-level" parameter controls during training—rather than just post-hoc patching—to explicitly steer models toward better generalization from the start.
Summary updated