TL;DR
This paper introduces Guided On-Policy Distillation (Guided-OPD), a novel method to efficiently transfer the capabilities of large multi-turn agents to smaller ones. By strategically mixing teacher and student-generated turns within a curriculum, Guided-OPD overcomes a key failure mode of existing methods, leading to significant improvements in task performance and efficiency.
What Problem Does This Paper Address?
Building powerful multi-turn agents that can plan, use tools, and interact with complex environments often requires massive language models. However, the inference cost of these large models makes them impractical for real-world deployment. On-Policy Distillation (OPD) offers a way to transfer these capabilities to smaller, more efficient "student" models. The challenge arises in multi-turn settings: small errors made by the student early in a sequence can compound, pushing the agent's trajectory outside the "familiar" state distribution of the large "teacher" model. This makes the teacher's guidance less reliable precisely when the student needs it most, hindering effective knowledge transfer.
Methods
The core innovation is Guided On-Policy Distillation (Guided-OPD). Instead of letting the student generate an entire trajectory independently, Guided-OPD introduces two key components:
- Turn-level Guidance: At each turn of a rollout, the agent's next action is sampled from either the teacher policy (with probability $\beta$) or the student policy (with probability $1-\beta$). This ensures that the agent stays closer to the teacher's familiar distribution, especially early in training. The paper also uses an asymmetric loss function: reverse KL divergence for student turns (to tighten the student's output to teacher modes) and forward KL divergence for teacher turns (to ensure the student covers teacher behaviors).
- Curriculum Guidance: The mixing probability $\beta$ is not fixed. It starts high at the beginning of training (strong teacher guidance) and gradually decays to zero over a predefined "curriculum stage." This allows the agent to gradually transition to a purely on-policy regime, mimicking the inference setting and avoiding a training-inference mismatch. A cosine decay schedule is used by default.
Datasets
The paper evaluates Guided-OPD on three established multi-turn agent benchmarks:
- ALFWorld: An embodied environment for household tasks.
- ScienceWorld: A text-based environment for scientific reasoning.
- WebShop: A simulated e-commerce environment for shopping tasks.
The authors distill Qwen3 students (0.6B, 1.7B, and 4B parameters) from a Qwen3-30B-A3B teacher model. No new benchmarks were introduced.
Key Insights and Ideas
- Rollout Drift is a Major Hurdle: The paper clearly demonstrates that in multi-turn agents, small student errors can lead to significant "rollout drift," where the agent deviates from the teacher's learned distribution, rendering teacher supervision ineffective. This is a critical insight for understanding OPD limitations in sequential decision-making.
- Turn-level Guidance Anchors Trajectories: Intervening at the turn level, rather than token level, aligns better with how state transitions occur in environments. This allows the teacher to "anchor" the agent's trajectory, preventing it from straying too far.
- Curriculum is Crucial for Transition: A decaying teacher intervention schedule is vital. Starting with strong guidance and gradually fading it allows the student to learn from reliable supervision initially and then develop its own independent capabilities, bridging the gap between training and inference.
- Smaller Students Benefit Most: The gains from Guided-OPD are more pronounced for smaller student models, as they are more prone to rollout drift and thus benefit more from the teacher's anchoring.
Key Results
Guided-OPD significantly outperforms vanilla OPD across the evaluated benchmarks:
- Average Gains: +21.1% in Score and +25.5% in Success Rate (SR) over vanilla OPD.
- Efficiency: Reduced interaction turns by an average of 6.0%.
- Smaller Students: For the smallest 0.6B student, gains reached +35.3% in Score and +32.8% in SR.
- Training Efficiency: Guided-OPD achieved the same quality as vanilla OPD in roughly 1/4 of the wall-clock training time, primarily due to shorter rollouts from the start.
- Ablation Studies: The asymmetric loss design and the curriculum schedule were shown to be critical components, with fixed mixing probabilities performing worse than the decaying curriculum.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Benchmarks: The evaluation was confined to three text-based environments. Future work could explore Guided-OPD in richer settings with different observation/action spaces (e.g., embodied agents with continuous control, tool-use agents with API calls).
- Intra-Family Distillation: The experiments focused on distilling within the same model family (Qwen3). Cross-family distillation might introduce challenges like tokenizer mismatches, requiring further investigation.
- Hyperparameter Sensitivity: While the default curriculum settings performed well, an adaptive schedule that reacts to teacher perplexity could potentially remove manual tuning and yield further gains.
- Scalability: The experiments were conducted with up to 4B student models and a 30B teacher. Testing on substantially larger scales and longer-horizon tasks is needed to fully understand the method's capabilities.
Summary updated