TL;DR
On-Policy Distillation (OPD) doesn't just transfer knowledge; it fundamentally reshapes how LLM agents interleave reasoning and action (ReAct patterns). While OPD can boost domain-specific accuracy, it often triggers "negative transfer"—like making an agent too "impatient" to wait for user input—which can be mitigated by selectively distilling only the tokens where a teacher model showed the most significant preference shifts during its own reinforcement learning.
What Problem Does This Paper Address?
Large Language Model (LLM) agents typically follow the ReAct (Reasoning + Acting) paradigm, where they "think" out loud before executing a tool call or interacting with an environment. To make these agents better, researchers use On-Policy Distillation (OPD): a student model generates trajectories, and a stronger teacher model provides feedback (supervision) on those specific paths.
However, we’ve noticed a "black box" effect. When we train an agent to be a specialist in one area (like solving math with code), it often starts behaving strangely in other areas (like household chores). This paper investigates how OPD reshapes these interaction patterns and why specialized training sometimes breaks an agent's "generic" ability to interact sensibly with humans and tools.
Methods
The authors systematically analyze OPD across four distinct domains:
- General Reasoning (GR): Pure natural-language math.
- Code-Interpretation (CI): Math using a Python interpreter.
- ALFWorld (ALF): Text-based embodied household tasks.
- $\tau^2$-bench: Complex conversational tool-use (e.g., airline/retail services).
To fix the "negative transfer" (where learning one task ruins another), they introduce Selective OPD (Signed-$\Delta$ Top95).
- The Logic: They compare a teacher model before and after it underwent domain-specific Reinforcement Learning (RL).
- The Technique: They calculate the difference ($\Delta$) in log-probabilities for every token. They then only distill the top 95% of tokens—the ones where the teacher's preference increased the most. This effectively filters out the "lowest-tail" tokens that represent surface-level task quirks or harmful over-specialization.
Datasets
The study utilizes several high-stakes benchmarks for evaluation:
- Mathematics: AIME2026 and BeyondAIME.
- Embodied AI: ALFWorld (140 in-distribution and 134 out-of-distribution tasks).
- Conversational Service: $\tau^2$-bench (100 tasks across airline, retail, and telecom sectors).
- Training Base: Qwen3-4B-Instruct-2507 served as the student model, while 30B-parameter variants were used as teachers.
Key Insights and Ideas
- The "Impatient Agent" Syndrome: Training on ALFWorld (where the environment responds instantly to every action) causes a catastrophic failure on $\tau^2$-bench (where the agent must wait for a user). The model becomes "impatient," calling tools before the user has even provided the necessary ID or authorization.
- Reasoning Route Shifts: OPD can fundamentally change how a model solves a problem. For example, training on pure math (GR) causes the model to stop using Python tools in CI tasks, even when the tool would be more reliable. It shifts from "computational" routes to "algebraic derivation" routes.
- Dependency Violations: In 55.8% of trajectories after ALFWorld training, agents attempted tool calls that violated user-information dependencies (e.g., trying to look up an account before the user gave them the account number).
- Generic vs. Task-Specific Abilities: The authors argue that agentic knowledge has two parts: generic abilities (how to reason/act) and task-specific patterns (habits of information acquisition). Negative transfer is usually caused by the student overfitting to the latter.
Key Results
- Catastrophic Drop: ALFWorld-only distillation caused $\tau^2$-bench accuracy to plummet from 41.5% to 6.6%.
- The Rescue: Using the Signed-$\Delta$ Top95 selective distillation recovered $\tau^2$ accuracy to 29.9% while still capturing 80% of the performance gains intended for the source task (ALFWorld).
- Math Efficiency: General-reasoning (GR) supervision reduced mathematical tool attempts from 70.4% to 14.4% but actually increased accuracy from 16.7% to 38.0%, showing the model learned to "think" better rather than just "code" more.
- Positive Transfer: Interestingly, Code-Interpreter (CI) training actually helped in ALFWorld and $\tau^2$-bench by reducing repeated actions and producing more complete queries.
Limitations and Future Work
- Model Scale: The study focused on the Qwen3-4B student family. It remains to be seen if larger models (70B+) exhibit the same degree of pattern reshaping or if they are more resilient to negative transfer.
- Protocol Sensitivity: The results might vary based on the specific interaction protocol (e.g., how tool calls are formatted).
- Future Directions: The authors suggest exploring more sophisticated ways to isolate "generic agentic logic" from "domain-specific habits" to create truly versatile multi-domain agents.
Summary based on "Agentic On-Policy Distillation Implicitly Reshapes LLMs' ReAct Patterns" (arXiv:2611.19244v1).
Summary updated