On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents

Gengsheng Li, Mao Zheng, Mingyang Song, Ruiqi Liu, Tianyu Yang, Jie Sun, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Dan Zhang, Jinqiao WangarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces Guided On-Policy Distillation (Guided-OPD), a novel method to efficiently transfer the capabilities of large multi-turn agents to smaller ones. By strategically mixing teacher and student-generated turns within a curriculum, Guided-OPD overcomes a key failure mode of existing methods, leading to significant improvements in task performance and efficiency.

What Problem Does This Paper Address?

Building powerful multi-turn agents that can plan, use tools, and interact with complex environments often requires massive language models. However, the inference cost of these large models makes them impractical for real-world deployment. On-Policy Distillation (OPD) offers a way to transfer these capabilities to smaller, more efficient "student" models. The challenge arises in multi-turn settings: small errors made by the student early in a sequence can compound, pushing the agent's trajectory outside the "familiar" state distribution of the large "teacher" model. This makes the teacher's guidance less reliable precisely when the student needs it most, hindering effective knowledge transfer.

Methods

The core innovation is Guided On-Policy Distillation (Guided-OPD). Instead of letting the student generate an entire trajectory independently, Guided-OPD introduces two key components:

  1. Turn-level Guidance: At each turn of a rollout, the agent's next action is sampled from either the teacher policy (with probability $\beta$) or the student policy (with probability $1-\beta$). This ensures that the agent stays closer to the teacher's familiar distribution, especially early in training. The paper also uses an asymmetric loss function: reverse KL divergence for student turns (to tighten the student's output to teacher modes) and forward KL divergence for teacher turns (to ensure the student covers teacher behaviors).
  2. Curriculum Guidance: The mixing probability $\beta$ is not fixed. It starts high at the beginning of training (strong teacher guidance) and gradually decays to zero over a predefined "curriculum stage." This allows the agent to gradually transition to a purely on-policy regime, mimicking the inference setting and avoiding a training-inference mismatch. A cosine decay schedule is used by default.

Datasets

The paper evaluates Guided-OPD on three established multi-turn agent benchmarks:

  • ALFWorld: An embodied environment for household tasks.
  • ScienceWorld: A text-based environment for scientific reasoning.
  • WebShop: A simulated e-commerce environment for shopping tasks.

The authors distill Qwen3 students (0.6B, 1.7B, and 4B parameters) from a Qwen3-30B-A3B teacher model. No new benchmarks were introduced.

Key Insights and Ideas

  • Rollout Drift is a Major Hurdle: The paper clearly demonstrates that in multi-turn agents, small student errors can lead to significant "rollout drift," where the agent deviates from the teacher's learned distribution, rendering teacher supervision ineffective. This is a critical insight for understanding OPD limitations in sequential decision-making.
  • Turn-level Guidance Anchors Trajectories: Intervening at the turn level, rather than token level, aligns better with how state transitions occur in environments. This allows the teacher to "anchor" the agent's trajectory, preventing it from straying too far.
  • Curriculum is Crucial for Transition: A decaying teacher intervention schedule is vital. Starting with strong guidance and gradually fading it allows the student to learn from reliable supervision initially and then develop its own independent capabilities, bridging the gap between training and inference.
  • Smaller Students Benefit Most: The gains from Guided-OPD are more pronounced for smaller student models, as they are more prone to rollout drift and thus benefit more from the teacher's anchoring.

Key Results

Guided-OPD significantly outperforms vanilla OPD across the evaluated benchmarks:

  • Average Gains: +21.1% in Score and +25.5% in Success Rate (SR) over vanilla OPD.
  • Efficiency: Reduced interaction turns by an average of 6.0%.
  • Smaller Students: For the smallest 0.6B student, gains reached +35.3% in Score and +32.8% in SR.
  • Training Efficiency: Guided-OPD achieved the same quality as vanilla OPD in roughly 1/4 of the wall-clock training time, primarily due to shorter rollouts from the start.
  • Ablation Studies: The asymmetric loss design and the curriculum schedule were shown to be critical components, with fixed mixing probabilities performing worse than the decaying curriculum.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Benchmarks: The evaluation was confined to three text-based environments. Future work could explore Guided-OPD in richer settings with different observation/action spaces (e.g., embodied agents with continuous control, tool-use agents with API calls).
  • Intra-Family Distillation: The experiments focused on distilling within the same model family (Qwen3). Cross-family distillation might introduce challenges like tokenizer mismatches, requiring further investigation.
  • Hyperparameter Sensitivity: While the default curriculum settings performed well, an adaptive schedule that reacts to teacher perplexity could potentially remove manual tuning and yield further gains.
  • Scalability: The experiments were conducted with up to 4B student models and a 30B teacher. Testing on substantially larger scales and longer-horizon tasks is needed to fully understand the method's capabilities.

Summary updated

Abstract

Multi-turn agents that plan, invoke tools, and interact with environments offer a promising paradigm for solving complex tasks, yet their capabilities typically rely on very large models whose inference cost is prohibitive in practice.On-Policy Distillation (OPD) is a natural recipe for transferring such capabilities to smaller students, but we find that it suffers a characteristic failure mode in this setting: small student errors compound across turns and push the trajectory out of the teacher's familiar state distribution, so the teacher's supervision becomes least reliable precisely where the student needs it most.We propose Guided On-Policy Distillation (Guided-OPD), a simple yet effective algorithm that mixes teacher- and student-generated turns within each rollout and schedules the teacher's intervention probability along a curriculum that decays to zero.Strong guidance keeps early trajectories close to the teacher distribution and is then gradually withdrawn to recover the purely on-policy regime used at inference.On ALFWorld, ScienceWorld, and WebShop, distilling Qwen3 students from a Qwen3-30B-A3B teacher, Guided-OPD improves Score by 21.1\% and Success Rate by 25.5\% over vanilla OPD on average, with larger gains on smaller students.
BibTeX
@misc{li2026policy,
  author = {Gengsheng Li and Mao Zheng and Mingyang Song and Ruiqi Liu and Tianyu Yang and Jie Sun and Qiyong Zhong and Haiyun Guo and Junfeng Fang and Dan Zhang and Jinqiao Wang},
  title = {{On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents}},
  year = {2026},
  eprint = {2606.15912},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.15912},
}
Figure