Abstract
Large Language Models (LLMs) acting as autonomous agents for software engineering tasks often rely on Test-Time Scaling (TTS)—sampling multiple solution trajectories and selecting the best one—to improve performance. However, standard alignment techniques like Direct Preference Optimization (DPO) frequently lead to "diversity collapse," where the model converges on a narrow set of outputs, rendering TTS ineffective. We present ENTROPO, a framework that incorporates entropy regularization into multi-turn preference optimization. By penalizing low-entropy policies, ENTROPO forces models to explore diverse, correct reasoning paths. Our results show that ENTROPO significantly enhances TTS efficiency, allowing a 30B parameter model to achieve top-tier performance on the SWE-bench leaderboard, outperforming much larger models.
Introduction
The use of LLMs as coding agents involves complex, multi-turn interactions with environments to resolve software issues. A common strategy to boost agentic performance is Test-Time Scaling (TTS), which involves generating $N$ parallel trajectories and using a selection mechanism to identify a successful patch. The effectiveness of TTS is fundamentally limited by the diversity of the generated samples. If a model consistently produces identical or highly similar trajectories, increasing $N$ yields diminishing returns.
The authors identify "diversity collapse" as a primary obstacle. Standard preference learning methods tend to concentrate probability mass on a single "winning" response. In the context of multi-turn workflows, this prevents the agent from exploring alternative tool-use sequences or hypotheses when initial attempts fail. ENTROPO addresses this by explicitly rewarding policy diversity during training.
The ENTROPO Framework
ENTROPO formulates multi-turn coding as an entropy-regularized Markov Decision Process (MDP). It introduces two novel loss functions: ENTROPO-DPO and ENTROPO-KTO. These objectives add an explicit entropy term, $\lambda H(\pi)$, which prevents the policy from becoming overly peaked.
Theoretical analysis reveals that ENTROPO acts as a form of inverse probability weighting. It assigns larger updates to high-utility trajectories that are underrepresented by the reference policy, effectively counteracting mode collapse and ensuring the model learns multiple ways to solve a problem.
At inference time, ENTROPO employs a hybrid trajectory selector. This system filters trajectories using model-free criteria (e.g., task completion, regression test passes) and a model-based verifier that scores the likelihood of success. Heuristics are also applied, such as preferring longer trajectories for complex, verified benchmarks to ensure thorough reasoning.
Experimental Evaluation
The framework was evaluated using several key datasets:
- SWE-Smith: Used for initial Supervised Fine-Tuning (SFT) to establish basic tool-use capabilities.
- R2E-Gym-subset: Used for preference learning and training the verifier model.
- SWE-bench Verified & Lite: The primary benchmarks consisting of real-world GitHub issues.
Results and Discussion
ENTROPO achieves state-of-the-art results among open-weight models. On SWE-bench Lite, a 30B model based on Qwen-Coder ranked 1st with a 49.7% resolve rate. On SWE-bench Verified, it ranked 4th overall with a 60.4% resolve rate, surpassed only by proprietary models or open-weight models (like 480B parameters) that are more than 10x larger.
A key finding is the improved scaling efficiency. As shown in the results, the performance gap between ENTROPO and standard M-DPO grows as the number of test-time samples $N$ increases. This confirms that entropy regularization successfully preserves the diversity necessary to fuel TTS. Quantitative analysis using cosine similarity further validated that ENTROPO-trained models maintain more diverse internal representations across different runs compared to baselines.
Conclusion
ENTROPO demonstrates that the failure of test-time scaling in coding agents is often a training-time issue rather than a search problem. By prioritizing trajectory diversity through entropy regularization, ENTROPO enables smaller models to compete with proprietary giants. Future work includes exploring online iterative settings and validating the framework in other complex reasoning domains beyond software engineering.
Summary updated