TL;DR
Communicative On-Policy Distillation (COPD) enables small language model agents to learn complex, long-horizon tasks by "talking" to powerful black-box teachers during training. By internalizing these natural-language interactions through on-policy self-distillation, the student models learn to solve tasks independently, achieving significant performance gains without needing access to the teacher's internal probability data (logits).
What Problem Does This Paper Address?
Training effective language model (LM) agents for long-horizon tasks—like building a website or solving complex coding bugs—is notoriously difficult. A common approach is On-Policy Distillation (OPD), where a smaller "student" model learns by mimicking a stronger "teacher."
However, traditional OPD has two major hurdles:
- The Logit Gap: Most state-of-the-art models (like GPT-4 or Claude) are "black boxes" that only provide text outputs. Traditional distillation requires "white-box" access to the teacher's internal token probabilities (logits) to calculate a learning signal.
- The "When" and "What" Problem: In long-horizon tasks, it's hard to pre-define exactly when a student needs help or what specific information it should ask for.
The authors propose that agents should learn more like humans: by asking for advice when stuck and then internalizing that advice so they can eventually work alone.
Methods
The core of the paper is Communicative On-Policy Distillation (COPD). Instead of comparing logits, the student uses natural language as the medium for distillation.
The process works in three main phases:
- Experience Collection: During training, the student acts in an environment. It can autonomously decide to send a free-form text request to a black-box adviser. The adviser's text reply is treated as a new observation that guides the student's next steps.
- Success Corpus: Only trajectories that result in a successful task completion and involved a consultation with the adviser are saved into a "Success Corpus."
- On-Policy Self-Distillation (OPSD): This is the "internalization" step. The student performs new "adviser-free" runs on the same tasks. It uses its own successful, adviser-aided trajectories from the corpus as "privileged" references. By comparing its current adviser-free attempt to the successful teacher-aided one, it generates a token-level learning signal to update its weights.
Crucially, the communication policy (when to ask) and the task policy (how to act) are learned jointly using Reinforcement Learning (PPO).
Datasets
The authors evaluated COPD across several challenging benchmarks:
- CAMG-II (Coding Agent Memory Gym II): The primary environment, featuring five domains: Shop, Coding, AutoResearch, General, and Visual tasks.
- Terminal-Bench 2.1: A benchmark for command-line interface agents.
- SWE-Bench Pro V2: A rigorous software engineering benchmark involving real-world GitHub issues.
Key Insights and Ideas
- Communication as a Policy Action: The student isn't given a template for questions. It learns to formulate its own requests and, more importantly, learns when the cost of asking is worth the benefit of the guidance.
- Internalization over Dependence: A major risk of "learning to ask" is that the student becomes a "prompt engineer" who can't function without the teacher. COPD specifically forces the student to practice without the adviser during the distillation phase to ensure the knowledge is internalized.
- The Value of "Consulted Success": The authors found that a success achieved with a teacher's help is a much stronger training signal than a "lucky" success achieved alone. As shown in Figure 4, a consulted success moves the needle significantly more than a self-success.
- Evolving Strategy: As training progresses, the student actually asks less frequently but asks better questions. It learns to save its "lifelines" for the most critical bottlenecks in a task.
Key Results
COPD demonstrated substantial improvements over existing black-box distillation methods:
- Superior Performance: On the CAMG-II benchmark, COPD-4B achieved a 59.8% average success rate, outperforming the strongest black-box baseline (ActionOPD) by 2.46 times the gain over the base model.
- Scaling and Generalization: A larger COPD-9B model reached a 61.3% success rate, nearly matching the performance of a much larger 35B parameter model (Qwen3.5-35B).
- Adviser-Free Success: The "internalization" was highly effective. While the initial student lost nearly 10 points of performance when the adviser was removed, the COPD-trained student's performance barely budged (only a 1.6-point drop), proving it had truly learned the underlying capabilities.
Limitations and Future Work
- Model Scale: The study focused on students up to 9B parameters. It remains to be seen if these communicative gains scale linearly to much larger models.
- Single Teacher: The current framework uses a single, frozen black-box teacher. Future work could explore "Multi-teacher OPD" (MOPD), where a student consults different specialists for different parts of a long-horizon task.
- Teacher Quality: The method assumes the teacher is significantly stronger than the student. If the teacher provides poor advice, the distillation process could potentially reinforce bad habits.
Summary updated