When a language model debugs its own code over several rounds of running tests and reading errors, it rarely fails because it cannot find the bug. It fails because each fix tends to damage something else. It deletes working lines, adds a special case that satisfies the visible test but not the real specification, or piles new logic into an already bloated function. The documented symptoms are consistent. Gains from automated debugging flatten after one to three rounds (CodeTransOcean). Test-driven refinement loops raise the rate of patches that pass local tests but fail hidden ones (Investigating Test Overfitting on SWE-bench). Code quality erodes across iterations in most agent trajectories (SlopCodeBench). Meanwhile, preference optimization for code trains models to prefer test-passing outputs over failing ones. It anchors training either to the base model's distribution or to whole correct programs. It does not anchor to the thing a repair should preserve, which is the parts of the current program that execution shows are already correct.
We propose training a multi-turn repair model with preference optimization in which the regularizer is tied to the program being repaired. Specifically, a one-sided penalty stops the model from becoming less willing than the base model to keep lines that passing tests exercise and failing tests do not. Preference labels are built from a test split the model never sees as feedback. A "chosen" repair is one that also passes held-out tests and breaks no previously passing test. The hardest "rejected" repairs are test-gaming patches and regressions. The bet is that the refinement plateau is largely a regression problem. If per-turn regressions and overfitting can be trained out, accuracy should keep climbing past turn three instead of flattening.
Core Claim
Direct preference optimization (DPO) trains a model so that it assigns relatively higher probability to a preferred output than to a dispreferred one, compared with a frozen copy of the starting model. Regularized variants add a second term that stops absolute probabilities from collapsing. CodeLutra adds supervised loss on the whole passing program. Localized Preference Optimization (LPO) applies supervised loss only to the tokens shared between chosen and rejected outputs. BAPO uses a one-sided hinge that fires only when the policy's confidence in a base response falls below the reference model's.
All three ask where probability mass should stay. In iterative repair there is a better answer than "near the base model" or "on whatever tokens the pair happens to share": it should stay on the code that execution says is working. We claim that an execution-weighted preservation anchor, combined with preference pairs that specifically punish regressions and test-gaming, will reduce per-turn regression and overfitting rates and extend multi-turn gains. Neither ingredient alone should achieve this, and sequence-level or shared-token anchoring should not either.
The objective for a repair turn has a context $x$ made of the task, the current program $p_t$, and the execution feedback. It is
$$\mathcal{L} = \mathcal{L}_{\text{DPO}}^{\text{div}}(y_w, y_l \mid x) + \lambda \sum_{\ell \in p_t \cap y_w} w_\ell \, \max\!\Big(0,\ \log \tfrac{\pi_{\text{ref}}(\ell \mid x)}{\pi_\theta(\ell \mid x)}\Big).$$
The symbols are as follows. $\mathcal{L}_{\text{DPO}}^{\text{div}}$ is the DPO loss restricted to tokens on which chosen repair $y_w$ and rejected repair $y_l$ differ, the masking used by IterPref and LPO. The sum runs over lines $\ell$ of $p_t$ that are kept in $y_w$, and $\pi_\theta(\ell \mid x)$ is the policy's probability of those line tokens in place. $\pi_{\text{ref}}$ is the frozen starting model, and $\lambda$ is a single strength coefficient. $w_\ell \in [0,1]$ is an execution-evidence weight: it is high for lines covered by passing tests and not by failing ones, and low for lines on failing paths. The hinge is one-sided, as in BAPO, so the model is never pushed toward more copying than the base model already does. It is only stopped from abandoning working code. This also limits the "α too high collapses to SFT" sensitivity noted for hybrid losses.
Research Program
The work has three parts. The first is the data pipeline. We start from problems with test suites, namely TACO/APPS training problems and the CodeFlow trace dataset from IterPref. We split each suite into visible tests, which are shown as feedback, and held-out tests, which are used only for labeling. We then roll out the current policy for up to five repair turns in a sandbox, sampling several candidate repairs per turn. For each candidate we record visible pass/fail, held-out pass/fail, which previously passing tests now fail, and line coverage. Coverage supplies $w_\ell$.
Pairs are formed within a turn from candidates of the same $p_t$. Chosen repairs pass visible and held-out tests with zero regressions; ties among chosen are broken by smaller edit size. Rejected repairs are drawn preferentially from two categories. The first is "gaming" candidates, which pass visible tests and fail held-out ones. The second is "regressing" candidates, which break a previously passing test. Ordinary visible-test failures fill in only when these are absent. Training runs for two or three on-policy rounds, regenerating rollouts from the updated model each round, in the style of the recursive data generation in ARIES. ARIES judged refinements with a learned reward model on text and math; here every label comes from executed tests.
The second part is the training method above, applied to Qwen2.5-Coder-7B-Instruct. This is the base used by both IterPref and Focused-DPO, so their reported numbers are a direct reference.
The third part is analysis. We decompose each multi-turn trajectory into turns that fixed something, turns that regressed something, and turns that gamed tests. This lets us test whether the plateau reported by CodeTransOcean is mostly regression rather than failure to localize. We also test whether the anchor changes how the model handles the trap documented in DebugBench, where tracebacks help on syntax bugs but hurt on logic bugs. Our hypothesis there is modest. Keeping working lines may reduce the "rewrite everything near the traceback" behavior, but it will not by itself supply algorithmic insight.
One prior result constrains the design. LPO's authors found that exactly one round of refinement was optimal when distilling secure-code ground truth, which suggests that multi-turn traces become noisy. We take this seriously by reporting results separately for pairs drawn from turn one and from later turns, and by dropping late-turn pairs if they hurt.
Baselines
The naive reference point is the instruct model run in the same five-turn self-debugging loop with no extra training and the same sampling budget. This shows how much any training buys.
The strongest existing training method is IterPref's Target-DPO. It applies DPO only to differing tokens of debugging edit pairs and reports 29.7% pass@1 on BigCodeBench-Hard for Qwen2.5-Coder-7B (IterPref). It is the closest competitor because it already uses debug traces and token localization; it lacks an anchor and held-out-aware labels. That 29.7% is a single-shot generation figure, so we will reproduce it under our harness and then also run IterPref's trained model inside our multi-turn loop. CodeLutra represents whole-program anchoring: DPO plus supervised loss on passing programs. The paper reports 76.6% Spider execution accuracy with Llama-3-8B (CodeLutra), but it has no reported multi-turn repair result, so we re-run it. LPO represents shared-token anchoring. Focused-DPO is a further single-turn reference, at 87.8% on HumanEval+ against 84.1% for the base.
Every trained baseline uses our exact rollout data, the same number of on-policy rounds, the same model and token budget, and the same hyperparameter search budget (an equal number of trials over β and the anchor coefficient). Inference-time rollback in the spirit of MEIC is run as an additional baseline. It reverts any turn that lowers the visible pass count and is applied on top of both the untrained model and the best trained baseline.
Ablations isolate the two ingredients. One replaces $w_\ell$ with uniform weights, which reduces to an LPO-like anchor on kept lines. Another removes the hinge and uses plain supervised loss on kept lines. A third replaces held-out-aware labels with visible-test-only labels.
Beating the field means three things. On DebugBench multi-bug instances and on multi-turn BigCodeBench-Hard and LiveCodeBench, our final-turn held-out pass rate must exceed the best baseline by more than seed variance, with three seeds. The per-turn regression rate and the gaming rate must both be lower. And the accuracy-versus-turn curve must still be rising at turns four and five where baselines are flat.
Why not just…?
Why not just add a supervised anchor to IterPref? This is the obvious combination, and the uniform-weight ablation tests it directly. If the uniform anchor matches the execution-weighted one, the contribution shrinks to "LPO applied to debug traces" and we will report it as such. We expect a gap for a specific reason. In repair, shared tokens between two candidate fixes include buggy lines that neither candidate touched. A uniform anchor reinforces those lines, while coverage weighting does not. The gap should therefore be largest on multi-bug DebugBench instances, where GPT-4 already falls to 55.9% (DebugBench).
Why not just roll back bad turns at inference, as MEIC does? This is a real threat for regressions, since rollback removes visible regressions for free. It cannot help with test-gaming, because at inference the gaming patch passes every test the system can see. The SWE-bench study found overfitting rising from 33.0% to 35.9% for GPT-4o under refinement precisely in this setting (Investigating Test Overfitting on SWE-bench). Rollback also does not make the next proposed fix any better. Running rollback on top of each trained model settles how much of our gain is additive to it. If rollback on the untrained model closes most of the gap, the training story is weak.
Why not just fine-tune on successful repair traces (rejection sampling)? Supervised training on passing repairs never shows the model a gaming or regressing patch as something to avoid. If held-out-aware filtering alone produced the effect, this baseline would capture it. We run it on the same chosen repairs. A result where it matches us on held-out pass rate but not on gaming rate, or the reverse, tells us which ingredient matters.
Why not just prompt for minimal edits? We include a "change as little as possible" instruction in the untrained baseline. SlopCodeBench's agents, which are prompted systems, still eroded code in 77% of trajectories, which suggests prompting alone is insufficient. We measure this rather than assume it.
Proposed Experiments
Evaluation uses four settings. The first is DebugBench, split by bug type and bug count. The second is HumanEval+ and MBPP+, where the original tests serve as visible feedback and the expanded tests are held out, a natural split that requires no new data. The third is BigCodeBench-Hard and LiveCodeBench in a five-turn loop with roughly half of each test suite visible. The fourth is SlopCodeBench as a transfer test of whether preservation-anchored training reduces verbosity and erosion under evolving specifications. The best agents there pass only 14.8% of checkpoints, so we report erosion and verbosity metrics alongside checkpoint pass rate. A stretch evaluation on SWE-bench Verified would measure the overfitting rate on the protocol of Ahmed et al., if compute allows.
Metrics are held-out pass@1 at each turn, per-turn regression rate, gaming rate (visible pass with held-out fail), edit size in lines, and the SlopCodeBench quality measures.
The decisive checks come in three predictions, each of which could fail. First, if the plateau is mostly regression, our method's improvement over IterPref should be small at turn one and grow over turns two through five. If the improvement is flat across turns, the plateau has a different cause. Second, the gaming rate under our method should fall below the untrained loop's rate, while every visible-test-only ablation should leave it unchanged or higher. If the held-out labels do not move the gaming rate, the overfitting claim is wrong. Third, execution-weighted anchoring should beat uniform anchoring specifically on multi-bug and logic-bug DebugBench instances. Parity there would mean the coverage signal adds nothing, and the simpler LPO-style anchor should be preferred.
References
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Investigating Test Overfitting on SWE-bench
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
- Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences
- BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
- IterPref: Focal Preference Learning for Code Generation via Iterative Debugging
- Evolving LLMs' Self-Refinement Capability via Synergistic Training-Inference Optimization
- Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points
- DebugBench: Evaluating Debugging Capability of Large Language Models
- MEIC: Re-thinking RTL Debug Automation using LLMs