SWE-OPD: On-Policy Distillation for Agentic Coding

Yuhang He, Yanyu Chen, Chuanzhe Guo, Jie Zhu, Zichen Tian, Hongyu Ge, Zixuan Ren, Huang He, Bingjin Chen, Junyuan Shang, Shuohuan Wang, Yongqi Zhang, Irwin King, Jing LiuarXiv 2026 · arXiv

TL;DR

SWE-OPD demonstrates that on-policy distillation (OPD) is a highly effective way to transfer complex, long-horizon software engineering capabilities from large "teacher" models to smaller "student" agents. By supervising the student's own interactions with executable code repositories, the authors achieve significant performance gains on SWE-bench and introduce an "offline" variant that slashes compute costs by up to 72% without sacrificing much accuracy.

What Problem Does This Paper Address?

Building autonomous coding agents is hard because software engineering is a "long-horizon" task. A single erroneous action early in a trajectory—like a bad file edit or a misunderstood test log—can fundamentally change the environment, leading to a cascade of failures.

Traditional Supervised Fine-Tuning (SFT) often fails here because it trains students on "perfect" teacher-generated trajectories. When the student inevitably makes a mistake at test time, it finds itself in a state it never saw during training and doesn't know how to recover. On-Policy Distillation (OPD) solves this by having the student "roll out" its own trajectories and having a teacher model provide feedback on the specific paths the student actually took. However, applying OPD to software engineering is computationally grueling due to the length of interaction histories and the need for live execution environments.

Methods

The authors introduce SWE-OPD, a framework for distilling agentic coding capabilities. The core setup involves a student model interacting with a real-world repository (using tools like ls, grep, and edit) while a larger teacher model provides token-level supervision.

The study explores two primary design axes:

  1. Online vs. Offline Trajectories:
    • Online OPD: The student policy is updated continuously, and new trajectories are sampled using the latest policy.
  2. Offline SWE-OPD: Trajectories are collected once from the initial student model. Teacher scores are precomputed and cached. This decouples the expensive environment interaction from the training loop.
  3. Loss Functions: They compare a Top-$K$ Reverse-KL (RKL) loss (which encourages the student to match the teacher's distribution for the most likely tokens) against a Sampled-Token (ST) policy-gradient loss.
Figure 1: Study design of SWE-OPD. Left: the student interacts with an executable repository; tool observations enter the conditioning context, while only student-generated tokens receive supervision. Middle & Right: The two design axes we compare are the trajectory source (current student in SWE-OPD vs. initial student in Offline SWE-OPD) and the teacher information used per position (top-K candidates in the RKL loss vs. the sampled token in the ST loss objective).

Notably, they also demonstrate that this works across different model families with different tokenizers by using a simple alignment method called GOLD to map teacher probabilities onto student tokens.

Datasets

  • Training: 5k queries from ScaleSWE and an internal dataset of ~4k tasks.
  • Evaluation: Three major software engineering benchmarks:
  • SWE-bench Verified: A subset of SWE-bench with human-verified task descriptions.
  • SWE-bench Pro: A more challenging version of the benchmark.
  • Multilingual SWE-bench: Testing cross-language coding capabilities.

Key Insights and Ideas

  • Trajectory Refresh is Optional: One of the most surprising findings is that "refreshing" trajectories (the "on-policy" part of OPD) isn't strictly necessary for improvement. The offline variant, which uses fixed trajectories from the base model, still significantly outperformed the baseline.
  • The "Repetition Trap": The authors discovered that a decreasing training loss doesn't always mean the agent is getting better. In some cases, specifically with the RKL loss on the GLM model, the student developed severe repetitive behavior even as the loss curve looked healthy.
  • The Support-Mass Gradient: They identified a mathematical "failure mechanism" in the Top-$K$ RKL loss. Because the loss only looks at a subset of tokens, the gradient can inadvertently push probability mass into "unscored" tail tokens, leading to model collapse or repetition.
Training dynamics showing that while loss (c) decreases, repetition (a) can spike and entropy (b) can collapse, signaling a failure in the distillation process.

Key Results

  • Performance Boost: Distilling Qwen3.6-27B into Qwen3.5-9B increased the resolve rate on SWE-bench Pro from 35.9% to 39.4%.
  • Efficiency Gains: The offline variant reduced end-to-end GPU-hours by 59.4% to 72.6% and halved the peak GPU memory requirement.
  • Cross-Tokenizer Success: Distilling GLM-4.5-Air from DeepSeek-V4-Flash-0731 (different tokenizers) raised the SWE-bench Pro resolve rate from 35.8% to 39.7%, a nearly 4% absolute improvement.
  • Ablation on $K$: They found that the size of the Top-$K$ support (from 128 to 1024) didn't significantly change the final performance, suggesting that a compact support of 128 tokens is sufficient for effective distillation.
Figure 2: Top-K ablation. Gains in percentage points.

Limitations and Future Work

The authors intentionally kept the study "minimal" to isolate the effects of OPD. Consequently, they did not combine distillation with Reinforcement Learning (RL) or use data filtering (e.g., only training on successful trajectories), both of which are standard in state-of-the-art agents.

Future work could involve:

  • Combining process-level distillation with outcome-level rewards (RL).
  • Investigating more sophisticated cross-vocabulary mapping for cross-tokenizer distillation.
  • Exploring whether OPD can be applied earlier in the model development cycle (e.g., during pre-training) to reduce costs.

Summary updated

Abstract

In this work, we study on-policy distillation (OPD) for transferring agentic coding capabilities from a large teacher model to a smaller student model. Prior work has focused primarily on reasoning and short-horizon agentic tasks. We present SWE-OPD, a minimal OPD study for software engineering, where the student interacts with executable repositories and the teacher supervises the prefixes visited by the student. Across three teacher--student pairs, two training sources, and three SWE benchmarks, we find that on-policy supervision transfers coding capability: supervision from Qwen3.6-27B increases the SWE-bench Pro resolve rate of Qwen3.5-9B from 35.9% to 39.4%. Refreshing trajectories is not strictly required either, as an offline variant with cached teacher scores still improves over the student baselines while cutting end-to-end GPU-hours by 59.4%--72.6% and halving peak GPU allocation, at benchmark-dependent accuracy. Furthermore, we extend the offline variant to corss-tokenizer scenarios: distilling GLM-4.5-Air from DeepSeek-V4-Flash-0731 raises its SWE-bench Pro and Multilingual resolve rate from 35.8% to 39.7% and from 41.8% to 46.2%, around 4% and 4.6% improvement, respectively. We also find that a decreasing loss, however, does not necessarily indicate improved agent performance: under top-$K$ reverse-KL loss, one student exhibits severe repetition despite a decreasing loss. We identify a support-mass gradient term as a potential mechanism underlying this failure.
BibTeX
@misc{he2026swe,
  author = {Yuhang He and Yanyu Chen and Chuanzhe Guo and Jie Zhu and Zichen Tian and Hongyu Ge and Zixuan Ren and Huang He and Bingjin Chen and Junyuan Shang and Shuohuan Wang and Yongqi Zhang and Irwin King and Jing Liu},
  title = {{SWE-OPD: On-Policy Distillation for Agentic Coding}},
  year = {2026},
  eprint = {2611.21020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21020},
}
Figure