Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization

Jiahao Yu, Zelei Cheng, Xian Wu, Xinyu Xing

Abstract

Large Language Models (LLMs) acting as autonomous agents for software engineering tasks often rely on Test-Time Scaling (TTS)—sampling multiple solution trajectories and selecting the best one—to improve performance. However, standard alignment techniques like Direct Preference Optimization (DPO) frequently lead to "diversity collapse," where the model converges on a narrow set of outputs, rendering TTS ineffective. We present ENTROPO, a framework that incorporates entropy regularization into multi-turn preference optimization. By penalizing low-entropy policies, ENTROPO forces models to explore diverse, correct reasoning paths. Our results show that ENTROPO significantly enhances TTS efficiency, allowing a 30B parameter model to achieve top-tier performance on the SWE-bench leaderboard, outperforming much larger models.

Introduction

The use of LLMs as coding agents involves complex, multi-turn interactions with environments to resolve software issues. A common strategy to boost agentic performance is Test-Time Scaling (TTS), which involves generating $N$ parallel trajectories and using a selection mechanism to identify a successful patch. The effectiveness of TTS is fundamentally limited by the diversity of the generated samples. If a model consistently produces identical or highly similar trajectories, increasing $N$ yields diminishing returns.

The authors identify "diversity collapse" as a primary obstacle. Standard preference learning methods tend to concentrate probability mass on a single "winning" response. In the context of multi-turn workflows, this prevents the agent from exploring alternative tool-use sequences or hypotheses when initial attempts fail. ENTROPO addresses this by explicitly rewarding policy diversity during training.

The ENTROPO Framework

ENTROPO formulates multi-turn coding as an entropy-regularized Markov Decision Process (MDP). It introduces two novel loss functions: ENTROPO-DPO and ENTROPO-KTO. These objectives add an explicit entropy term, $\lambda H(\pi)$, which prevents the policy from becoming overly peaked.

The ENTROPO framework for multi-turn coding agents.

Theoretical analysis reveals that ENTROPO acts as a form of inverse probability weighting. It assigns larger updates to high-utility trajectories that are underrepresented by the reference policy, effectively counteracting mode collapse and ensuring the model learns multiple ways to solve a problem.

At inference time, ENTROPO employs a hybrid trajectory selector. This system filters trajectories using model-free criteria (e.g., task completion, regression test passes) and a model-based verifier that scores the likelihood of success. Heuristics are also applied, such as preferring longer trajectories for complex, verified benchmarks to ensure thorough reasoning.

Experimental Evaluation

The framework was evaluated using several key datasets:

  • SWE-Smith: Used for initial Supervised Fine-Tuning (SFT) to establish basic tool-use capabilities.
  • R2E-Gym-subset: Used for preference learning and training the verifier model.
  • SWE-bench Verified & Lite: The primary benchmarks consisting of real-world GitHub issues.

Results and Discussion

ENTROPO achieves state-of-the-art results among open-weight models. On SWE-bench Lite, a 30B model based on Qwen-Coder ranked 1st with a 49.7% resolve rate. On SWE-bench Verified, it ranked 4th overall with a 60.4% resolve rate, surpassed only by proprietary models or open-weight models (like 480B parameters) that are more than 10x larger.

Test-time scaling performance on SWE-bench-Verified.

A key finding is the improved scaling efficiency. As shown in the results, the performance gap between ENTROPO and standard M-DPO grows as the number of test-time samples $N$ increases. This confirms that entropy regularization successfully preserves the diversity necessary to fuel TTS. Quantitative analysis using cosine similarity further validated that ENTROPO-trained models maintain more diverse internal representations across different runs compared to baselines.

Conclusion

ENTROPO demonstrates that the failure of test-time scaling in coding agents is often a training-time issue rather than a search problem. By prioritizing trajectory diversity through entropy regularization, ENTROPO enables smaller models to compete with proprietary giants. Future work includes exploring online iterative settings and validating the framework in other complex reasoning domains beyond software engineering.

Summary updated

Abstract

Software engineering presents complex, multi-step challenges for Large Language Models (LLMs), requiring reasoning over large codebases and coordinated tool use. The difficulty of these tasks is exemplified by benchmarks like SWEBENCH, where current LLMs still struggle to resolve real-world issues. A promising approach to enhance performance is test-time scaling (TTS), but its gains are heavily dependent on the diversity of model outputs. While standard alignment methods such as Direct Preference Optimization (DPO) and Kahneman-Tversky Optimization (KTO) are effective at aligning model outputs with human preferences, this process can come at the cost of reduced diversity, limiting the effectiveness of TTS. Additionally, existing preference optimization algorithms are typically designed for single-turn tasks and do not fully address the complexities of multi-turn reasoning and tool integration required for interactive coding agents. To bridge this gap, we introduce ENTROPO, an entropy-enhanced framework that adapts existing preference optimization algorithms to the multi-turn, tool-assisted setting. ENTROPO augments the preference objective to explicitly preserve policy entropy and generalizes learning to optimize over multi-turn interactions rather than single-turn responses. We validate ENTROPO by fine-tuning a diverse suite of models from different families and sizes (up to 106B parameters). To maximize performance gains from TTS, we further propose a hybrid best-trajectory selection scheme combining a learned verifier model with model free approaches. On the SWEBENCH leaderboard, our approach establishes new state-of-the-art results among open-weight models. A 30B parameter model trained with ENTROPO ranks 1st on SWEBENCH-LITE and 4th on SWEBENCH-VERIFIED on the open-weight leaderboard, surpassed only by models with over 10x more parameters(e.g., >350B). These results highlight the importance of preserving diversity for effective test-time scaling and establish ENTROPO as a robust method for building powerful, interactive coding agents.
BibTeX
@misc{yu2025building,
  author = {Jiahao Yu and Zelei Cheng and Xian Wu and Xinyu Xing},
  title = {{Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization}},
  year = {2025},
  eprint = {2509.12434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.12434},
}
Figure