TL;DR
Standard alignment techniques often suffer from "diversity collapse," which severely handicaps test-time scaling (TTS) because parallel rollouts end up repeating identical mistakes. To fix this, the authors propose ENTROPO, an entropy-regularized preference optimization framework extended to multi-turn, tool-assisted coding agents, paired with a robust hybrid trajectory selector. Using ENTROPO and test-time compute, an open-weight 30B model achieves state-of-the-art results on SWE-bench Lite (49.3%) and Verified (59.4%), outperforming online RL methods and rivaling models over 10× its size.
What Problem Does This Paper Address?
Complex software engineering (SWE) tasks require multi-turn planning, codebase navigation, code editing, and interpreting execution feedback. A natural way to boost agent performance is test-time scaling (TTS)—generating multiple candidate solution trajectories in parallel and picking the best one.
However, TTS only works if sampled trajectories explore distinct hypotheses and solutions. Common preference optimization algorithms (like DPO and KTO) inadvertently trigger diversity collapse, concentrating probability mass onto a narrow set of outputs. When an aligned model samples redundant rollouts, the marginal returns of extra compute quickly vanish. Furthermore, existing preference optimization objectives are almost exclusively derived for single-turn text completions, leaving the sequential, tool-mediated nature of software engineering agents unaddressed.
Methods
ENTROPO bridges single-turn preference optimization and multi-turn agentic workflows through two core contributions:
- Entropy-Regularized Multi-Turn Preference Optimization:
- Grounded in maximum causal entropy Markov Decision Processes (MDPs), ENTROPO decouples the standard KL divergence penalty into a cross-entropy term against the reference policy $\pi_{\text{ref}}$ and an explicit policy entropy bonus $\alpha H(\pi)$.
- The authors mathematically derive closed-form optimal policies across multi-turn trajectories using backward induction, extending the framework to both pairwise (ENTROPO-DPO) and binary-feedback (ENTROPO-KTO) formulations.
- Training follows a two-stage regimen: Supervised Fine-Tuning (SFT) on teacher demonstrations to learn the tool scaffold (R2E framework), followed by ENTROPO fine-tuning using trajectories labeled by unit-test pass/fail signals.
- Hybrid Best-Trajectory Selector:
- At inference, the agent samples $N=16$ parallel trajectories. Instead of relying solely on an imperfect neural reward model, ENTROPO filters candidates hierarchically:
- Discard truncated trajectories (step/token limits).
- Discard trajectories failing existing repository regression tests.
- Apply a conservative probability threshold ($\ge 0.01$) from a trained verifier model.
- Select among remaining candidates using step-count heuristics (e.g., favoring longer trajectories on verified benchmarks to reward thorough testing).
Datasets
The authors evaluate and train using several open SWE benchmarks:
- SFT Training: SWE-Smith (8,736 Python instances with clean problem statements).
- Preference & Verifier Training: R2E-Gym-subset (4,578 instances with execution oracles; shares no repository overlap with evaluation sets).
- Evaluation: Standard official test sets of SWE-bench Verified (500 instances) and SWE-bench Lite (300 instances).
Key Insights and Ideas
- Temperature is not entropy: Simply cranking up the sampling temperature during inference degraded performance beyond $T=0.9$. Higher temperatures introduce erratic token noise that breaks syntax and tool-calling precision, whereas ENTROPO preserves structural trajectory diversity while maintaining high per-step execution quality.
- Offline preference learning can rival online RL: ENTROPO-KTO+TTS outperformed online RL-based methods like DeepSWE-TTS (59.4% vs. 58.8% on Verified) despite requiring substantially less infrastructure and training compute.
- Verifier as a filter, not a ranker: Learned verifiers in complex coding tasks are prone to false positives and calibration issues. Using the verifier strictly as a loose rejector ($\ge 0.01$) and letting rule-based heuristics pick the final candidate yielded better results than ranking directly by verifier score.
- Benchmark noise dictates trajectory heuristics: On human-vetted problems (SWE-bench Verified), longer trajectories indicate thorough exploration and testing. On noisier, underspecified benchmarks (SWE-bench Lite), shorter trajectories perform better by avoiding hallucinated complexity from ambiguous issue descriptions.
Key Results
- Leaderboard State-of-the-Art: Qwen3-Coder-30B trained with ENTROPO-KTO + TTS achieved 59.4% on SWE-bench Verified and 49.3% on SWE-bench Lite. It ranks 1st on SWE-bench Lite among open-weight models (beating 72B and 671B models) and 4th on Verified, trailing only massive proprietary mixtures-of-experts (e.g., Qwen3-Coder 480B, Kimi K2 1T, GLM-4.5 355B).
- Consistent TTS Scaling: Unlike standard multi-turn KTO and DPO—where resolve rates plateau as candidate samples grow—ENTROPO steadily widened its lead as $N$ scaled from 1 to 16.
- Enabling Small Models: ENTROPO dramatically resurrected small models. Qwen3-4B went from near-zero utility after SFT (1.7% on Verified / 1.2% on Lite) to over 11.5% with ENTROPO and TTS.
Limitations and Future Work
- Compute Budgets: Parallel sampling was evaluated only up to $N=16$ rollouts due to resource limits; exploring larger search budgets remains open.
- Heuristic Reliance: The trajectory selection scheme relies partly on empirical heuristics (step length) rather than theoretically grounded stopping criteria or learned search policies.
- Offline vs. Online: ENTROPO is currently implemented offline. Extending entropy-regularized diversity incentives into an online reinforcement learning framework could yield even stronger multi-turn policies.
- Search Architecture: Rollouts were generated independently end-to-end; combining ENTROPO with structured intermediate search (such as Monte Carlo Tree Search or patch merging) is a promising next step.
Summary updated