TL;DR
SWE-OPD demonstrates that on-policy distillation (OPD) is a highly effective way to transfer complex, long-horizon software engineering capabilities from large "teacher" models to smaller "student" agents. By supervising the student's own interactions with executable code repositories, the authors achieve significant performance gains on SWE-bench and introduce an "offline" variant that slashes compute costs by up to 72% without sacrificing much accuracy.
What Problem Does This Paper Address?
Building autonomous coding agents is hard because software engineering is a "long-horizon" task. A single erroneous action early in a trajectory—like a bad file edit or a misunderstood test log—can fundamentally change the environment, leading to a cascade of failures.
Traditional Supervised Fine-Tuning (SFT) often fails here because it trains students on "perfect" teacher-generated trajectories. When the student inevitably makes a mistake at test time, it finds itself in a state it never saw during training and doesn't know how to recover. On-Policy Distillation (OPD) solves this by having the student "roll out" its own trajectories and having a teacher model provide feedback on the specific paths the student actually took. However, applying OPD to software engineering is computationally grueling due to the length of interaction histories and the need for live execution environments.
Methods
The authors introduce SWE-OPD, a framework for distilling agentic coding capabilities. The core setup involves a student model interacting with a real-world repository (using tools like ls, grep, and edit) while a larger teacher model provides token-level supervision.
The study explores two primary design axes:
- Online vs. Offline Trajectories:
- Online OPD: The student policy is updated continuously, and new trajectories are sampled using the latest policy.
- Offline SWE-OPD: Trajectories are collected once from the initial student model. Teacher scores are precomputed and cached. This decouples the expensive environment interaction from the training loop.
- Loss Functions: They compare a Top-$K$ Reverse-KL (RKL) loss (which encourages the student to match the teacher's distribution for the most likely tokens) against a Sampled-Token (ST) policy-gradient loss.
Notably, they also demonstrate that this works across different model families with different tokenizers by using a simple alignment method called GOLD to map teacher probabilities onto student tokens.
Datasets
- Training: 5k queries from ScaleSWE and an internal dataset of ~4k tasks.
- Evaluation: Three major software engineering benchmarks:
- SWE-bench Verified: A subset of SWE-bench with human-verified task descriptions.
- SWE-bench Pro: A more challenging version of the benchmark.
- Multilingual SWE-bench: Testing cross-language coding capabilities.
Key Insights and Ideas
- Trajectory Refresh is Optional: One of the most surprising findings is that "refreshing" trajectories (the "on-policy" part of OPD) isn't strictly necessary for improvement. The offline variant, which uses fixed trajectories from the base model, still significantly outperformed the baseline.
- The "Repetition Trap": The authors discovered that a decreasing training loss doesn't always mean the agent is getting better. In some cases, specifically with the RKL loss on the GLM model, the student developed severe repetitive behavior even as the loss curve looked healthy.
- The Support-Mass Gradient: They identified a mathematical "failure mechanism" in the Top-$K$ RKL loss. Because the loss only looks at a subset of tokens, the gradient can inadvertently push probability mass into "unscored" tail tokens, leading to model collapse or repetition.
Key Results
- Performance Boost: Distilling Qwen3.6-27B into Qwen3.5-9B increased the resolve rate on SWE-bench Pro from 35.9% to 39.4%.
- Efficiency Gains: The offline variant reduced end-to-end GPU-hours by 59.4% to 72.6% and halved the peak GPU memory requirement.
- Cross-Tokenizer Success: Distilling GLM-4.5-Air from DeepSeek-V4-Flash-0731 (different tokenizers) raised the SWE-bench Pro resolve rate from 35.8% to 39.7%, a nearly 4% absolute improvement.
- Ablation on $K$: They found that the size of the Top-$K$ support (from 128 to 1024) didn't significantly change the final performance, suggesting that a compact support of 128 tokens is sufficient for effective distillation.
Limitations and Future Work
The authors intentionally kept the study "minimal" to isolate the effects of OPD. Consequently, they did not combine distillation with Reinforcement Learning (RL) or use data filtering (e.g., only training on successful trajectories), both of which are standard in state-of-the-art agents.
Future work could involve:
- Combining process-level distillation with outcome-level rewards (RL).
- Investigating more sophisticated cross-vocabulary mapping for cross-tokenizer distillation.
- Exploring whether OPD can be applied earlier in the model development cycle (e.g., during pre-training) to reduce costs.
Summary updated