Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization

Jiahao Yu, Zelei Cheng, Xian Wu, Xinyu XingSubmission to ICLR 2026 · Rejected

TL;DR

Standard alignment techniques often suffer from "diversity collapse," which severely handicaps test-time scaling (TTS) because parallel rollouts end up repeating identical mistakes. To fix this, the authors propose ENTROPO, an entropy-regularized preference optimization framework extended to multi-turn, tool-assisted coding agents, paired with a robust hybrid trajectory selector. Using ENTROPO and test-time compute, an open-weight 30B model achieves state-of-the-art results on SWE-bench Lite (49.3%) and Verified (59.4%), outperforming online RL methods and rivaling models over 10× its size.

What Problem Does This Paper Address?

Complex software engineering (SWE) tasks require multi-turn planning, codebase navigation, code editing, and interpreting execution feedback. A natural way to boost agent performance is test-time scaling (TTS)—generating multiple candidate solution trajectories in parallel and picking the best one.

However, TTS only works if sampled trajectories explore distinct hypotheses and solutions. Common preference optimization algorithms (like DPO and KTO) inadvertently trigger diversity collapse, concentrating probability mass onto a narrow set of outputs. When an aligned model samples redundant rollouts, the marginal returns of extra compute quickly vanish. Furthermore, existing preference optimization objectives are almost exclusively derived for single-turn text completions, leaving the sequential, tool-mediated nature of software engineering agents unaddressed.

Methods

ENTROPO bridges single-turn preference optimization and multi-turn agentic workflows through two core contributions:

  1. Entropy-Regularized Multi-Turn Preference Optimization:
  2. Grounded in maximum causal entropy Markov Decision Processes (MDPs), ENTROPO decouples the standard KL divergence penalty into a cross-entropy term against the reference policy $\pi_{\text{ref}}$ and an explicit policy entropy bonus $\alpha H(\pi)$.
  3. The authors mathematically derive closed-form optimal policies across multi-turn trajectories using backward induction, extending the framework to both pairwise (ENTROPO-DPO) and binary-feedback (ENTROPO-KTO) formulations.
  4. Training follows a two-stage regimen: Supervised Fine-Tuning (SFT) on teacher demonstrations to learn the tool scaffold (R2E framework), followed by ENTROPO fine-tuning using trajectories labeled by unit-test pass/fail signals.
  5. Hybrid Best-Trajectory Selector:
  6. At inference, the agent samples $N=16$ parallel trajectories. Instead of relying solely on an imperfect neural reward model, ENTROPO filters candidates hierarchically:
  7. Discard truncated trajectories (step/token limits).
  8. Discard trajectories failing existing repository regression tests.
  9. Apply a conservative probability threshold ($\ge 0.01$) from a trained verifier model.
  10. Select among remaining candidates using step-count heuristics (e.g., favoring longer trajectories on verified benchmarks to reward thorough testing).

Datasets

The authors evaluate and train using several open SWE benchmarks:

  • SFT Training: SWE-Smith (8,736 Python instances with clean problem statements).
  • Preference & Verifier Training: R2E-Gym-subset (4,578 instances with execution oracles; shares no repository overlap with evaluation sets).
  • Evaluation: Standard official test sets of SWE-bench Verified (500 instances) and SWE-bench Lite (300 instances).

Key Insights and Ideas

  • Temperature is not entropy: Simply cranking up the sampling temperature during inference degraded performance beyond $T=0.9$. Higher temperatures introduce erratic token noise that breaks syntax and tool-calling precision, whereas ENTROPO preserves structural trajectory diversity while maintaining high per-step execution quality.
  • Offline preference learning can rival online RL: ENTROPO-KTO+TTS outperformed online RL-based methods like DeepSWE-TTS (59.4% vs. 58.8% on Verified) despite requiring substantially less infrastructure and training compute.
  • Verifier as a filter, not a ranker: Learned verifiers in complex coding tasks are prone to false positives and calibration issues. Using the verifier strictly as a loose rejector ($\ge 0.01$) and letting rule-based heuristics pick the final candidate yielded better results than ranking directly by verifier score.
  • Benchmark noise dictates trajectory heuristics: On human-vetted problems (SWE-bench Verified), longer trajectories indicate thorough exploration and testing. On noisier, underspecified benchmarks (SWE-bench Lite), shorter trajectories perform better by avoiding hallucinated complexity from ambiguous issue descriptions.

Key Results

  • Leaderboard State-of-the-Art: Qwen3-Coder-30B trained with ENTROPO-KTO + TTS achieved 59.4% on SWE-bench Verified and 49.3% on SWE-bench Lite. It ranks 1st on SWE-bench Lite among open-weight models (beating 72B and 671B models) and 4th on Verified, trailing only massive proprietary mixtures-of-experts (e.g., Qwen3-Coder 480B, Kimi K2 1T, GLM-4.5 355B).
  • Consistent TTS Scaling: Unlike standard multi-turn KTO and DPO—where resolve rates plateau as candidate samples grow—ENTROPO steadily widened its lead as $N$ scaled from 1 to 16.
  • Enabling Small Models: ENTROPO dramatically resurrected small models. Qwen3-4B went from near-zero utility after SFT (1.7% on Verified / 1.2% on Lite) to over 11.5% with ENTROPO and TTS.

Limitations and Future Work

  • Compute Budgets: Parallel sampling was evaluated only up to $N=16$ rollouts due to resource limits; exploring larger search budgets remains open.
  • Heuristic Reliance: The trajectory selection scheme relies partly on empirical heuristics (step length) rather than theoretically grounded stopping criteria or learned search policies.
  • Offline vs. Online: ENTROPO is currently implemented offline. Extending entropy-regularized diversity incentives into an online reinforcement learning framework could yield even stronger multi-turn policies.
  • Search Architecture: Rollouts were generated independently end-to-end; combining ENTROPO with structured intermediate search (such as Monte Carlo Tree Search or patch merging) is a promising next step.

Summary updated

Abstract

Software engineering presents complex, multi-step challenges for Large Language Models (LLMs), requiring reasoning over large codebases and coordinated tool use. The difficulty of these tasks is exemplified by benchmarks like SWEBENCH, where current LLMs still struggle to resolve real-world issues. A promising approach to enhance performance is test-time scaling (TTS), but its gains are heavily dependent on the diversity of model outputs. While standard alignment methods such as Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) are effective at aligning model outputs with human preferences, this process can come at the cost of reduced diversity, limiting the effectiveness of TTS. Additionally, existing preference optimization algorithms are typically designed for single-turn tasks and do not fully address the complexities of multi-turn reasoning and tool integration required for interactive coding agents. To bridge this gap, we introduce ENTROPO, an entropy-enhanced framework that adapts existing preference optimization algorithms to the multi-turn, tool-assisted setting. ENTROPO augments the preference objective to explicitly preserve policy entropy and generalizes learning to optimize over multi-turn interactions rather than single-turn responses. We validate ENTROPO by fine-tuning a diverse suite of models from different families and sizes (up to 106B parameters). To maximize performance gains from TTS, we further propose a hybrid best-trajectory selection scheme combining a learned verifier model with model free approaches. On the SWEBENCH leaderboard, our approach establishes new state-of-the-art results among open-weight models. A 30B parameter model trained with ENTROPO ranks 1st on SWEBENCH-LITE and 4th on SWEBENCH-VERIFIED on the open-weight leaderboard, surpassed only by models with over 10x more parameters(e.g., >350B). These results highlight the importance of preserving diversity for effective test-time scaling and establish ENTROPO as a robust method for building powerful, interactive coding agents.
BibTeX
@misc{yu2025building,
  author = {Jiahao Yu and Zelei Cheng and Xian Wu and Xinyu Xing},
  title = {{Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization}},
  year = {2025},
  eprint = {2509.12434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12434},
}
Figure