Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks

Lirui Luo, Kelong Mao, Xinwei Yang, Rongqing Li, Luyu Chen, Kieran Wong, Yudong Guo, Xinrui Wang, Jiayin Zhu, Gu Simiu, Sulong Xu, Cong FangarXiv 2026 · arXiv

TL;DR

Communicative On-Policy Distillation (COPD) enables small language model agents to learn complex, long-horizon tasks by "talking" to powerful black-box teachers during training. By internalizing these natural-language interactions through on-policy self-distillation, the student models learn to solve tasks independently, achieving significant performance gains without needing access to the teacher's internal probability data (logits).

What Problem Does This Paper Address?

Training effective language model (LM) agents for long-horizon tasks—like building a website or solving complex coding bugs—is notoriously difficult. A common approach is On-Policy Distillation (OPD), where a smaller "student" model learns by mimicking a stronger "teacher."

However, traditional OPD has two major hurdles:

  1. The Logit Gap: Most state-of-the-art models (like GPT-4 or Claude) are "black boxes" that only provide text outputs. Traditional distillation requires "white-box" access to the teacher's internal token probabilities (logits) to calculate a learning signal.
  2. The "When" and "What" Problem: In long-horizon tasks, it's hard to pre-define exactly when a student needs help or what specific information it should ask for.

The authors propose that agents should learn more like humans: by asking for advice when stuck and then internalizing that advice so they can eventually work alone.

Methods

The core of the paper is Communicative On-Policy Distillation (COPD). Instead of comparing logits, the student uses natural language as the medium for distillation.

Overview of the COPD framework, showing how student-teacher communication is distilled into an adviser-free policy.

The process works in three main phases:

  • Experience Collection: During training, the student acts in an environment. It can autonomously decide to send a free-form text request to a black-box adviser. The adviser's text reply is treated as a new observation that guides the student's next steps.
  • Success Corpus: Only trajectories that result in a successful task completion and involved a consultation with the adviser are saved into a "Success Corpus."
  • On-Policy Self-Distillation (OPSD): This is the "internalization" step. The student performs new "adviser-free" runs on the same tasks. It uses its own successful, adviser-aided trajectories from the corpus as "privileged" references. By comparing its current adviser-free attempt to the successful teacher-aided one, it generates a token-level learning signal to update its weights.

Crucially, the communication policy (when to ask) and the task policy (how to act) are learned jointly using Reinforcement Learning (PPO).

Datasets

The authors evaluated COPD across several challenging benchmarks:

  • CAMG-II (Coding Agent Memory Gym II): The primary environment, featuring five domains: Shop, Coding, AutoResearch, General, and Visual tasks.
  • Terminal-Bench 2.1: A benchmark for command-line interface agents.
  • SWE-Bench Pro V2: A rigorous software engineering benchmark involving real-world GitHub issues.

Key Insights and Ideas

  • Communication as a Policy Action: The student isn't given a template for questions. It learns to formulate its own requests and, more importantly, learns when the cost of asking is worth the benefit of the guidance.
  • Internalization over Dependence: A major risk of "learning to ask" is that the student becomes a "prompt engineer" who can't function without the teacher. COPD specifically forces the student to practice without the adviser during the distillation phase to ensure the knowledge is internalized.
  • The Value of "Consulted Success": The authors found that a success achieved with a teacher's help is a much stronger training signal than a "lucky" success achieved alone. As shown in Figure 4, a consulted success moves the needle significantly more than a self-success.
Comparison showing that successes derived from consultations provide a stronger learning signal than unassisted successes.
  • Evolving Strategy: As training progresses, the student actually asks less frequently but asks better questions. It learns to save its "lifelines" for the most critical bottlenecks in a task.

Key Results

COPD demonstrated substantial improvements over existing black-box distillation methods:

  • Superior Performance: On the CAMG-II benchmark, COPD-4B achieved a 59.8% average success rate, outperforming the strongest black-box baseline (ActionOPD) by 2.46 times the gain over the base model.
  • Scaling and Generalization: A larger COPD-9B model reached a 61.3% success rate, nearly matching the performance of a much larger 35B parameter model (Qwen3.5-35B).
  • Adviser-Free Success: The "internalization" was highly effective. While the initial student lost nearly 10 points of performance when the adviser was removed, the COPD-trained student's performance barely budged (only a 1.6-point drop), proving it had truly learned the underlying capabilities.
Main results table showing COPD attaining the highest average success rates across various domains compared to baselines.

Limitations and Future Work

  • Model Scale: The study focused on students up to 9B parameters. It remains to be seen if these communicative gains scale linearly to much larger models.
  • Single Teacher: The current framework uses a single, frozen black-box teacher. Future work could explore "Multi-teacher OPD" (MOPD), where a student consults different specialists for different parts of a long-horizon task.
  • Teacher Quality: The method assumes the teacher is significantly stronger than the student. If the teacher provides poor advice, the distillation process could potentially reinforce bad habits.

Summary updated

Abstract

On-policy distillation (OPD) is a powerful paradigm for improving language-model agents on long-horizon tasks, yet its conventional formulation relies on teacher logits that text-only black- box models do not expose. To address this limitation, we introduce Communicative On-Policy Distillation (COPD), which uses natural-language communication as a medium for distilling a stronger model's capabilities into a smaller student. During task execution, the student au- tonomously decides when to consult a black-box teacher and formulates open-ended natural- language requests without predefined question templates or intervention schedules. Successful student-teacher interactions are retained as privileged references to supervise fresh adviser-free student rollouts through on-policy self-distillation, jointly with task-reward reinforcement learning. On CAMG-II, COPD achieves 2.46 times the gain of the strongest black-box OPD baseline. The capabilities it learns through communication generalize to Terminal-Bench 2.1 and SWE-Bench Pro V2, improving task success without adviser access.
BibTeX
@misc{luo2026communicative,
  author = {Lirui Luo and Kelong Mao and Xinwei Yang and Rongqing Li and Luyu Chen and Kieran Wong and Yudong Guo and Xinrui Wang and Jiayin Zhu and Gu Simiu and Sulong Xu and Cong Fang},
  title = {{Communicative On-Policy Distillation: Distilling Black-Box Teacher Capabilities through Communication for Long-Horizon Tasks}},
  year = {2026},
  eprint = {2611.23673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23673},
}
Figure