When a language model writes a program that fails its tests, standard practice is to show it the error and ask for a fix, then repeat. Each round uses the same kind of repair, whether the failure is a typo, an off-by-one boundary, a timeout that calls for a different algorithm, or a misread specification. These failures need different kinds of thinking. A one-line patch needs no deliberation. A timeout usually means the program should be redesigned. A wrong answer on a small input often calls for tracing the execution by hand. We propose to treat each refinement round as a choice among a small set of distinct repair strategies. A lightweight learned selector makes that choice by reading the concrete execution feedback from the last attempt. We will test whether this beats both fixed repair loops and simply restarting with fresh samples when all methods get the same token budget.
The opportunity is real because two literatures have each built half of this. Work on reasoning-strategy selection shows that choosing among chain-of-thought, program-style reasoning, and decomposition per problem helps. It makes that choice once, before any code runs. Work on iterative code repair shows that loops with execution feedback help but plateau after one to three rounds. It uses a single repair template throughout. Our question is whether the plateau comes partly from that uniformity, and whether the feedback itself carries enough signal to pick the right next move.
Core Claim
Execution feedback is informative about which repair strategy will succeed next. Examples of such feedback are the error class, the fraction of tests passing, whether the previous edit changed the failure, and how many rounds have passed. The claim has two parts, and the second is only worth pursuing if the first holds. First, at a failing state, the best repair strategy varies across states, and it varies enough that an oracle picking per round would substantially outperform the best single fixed strategy. Second, a cheap selector trained on counterfactual rollouts can recover a meaningful share of that oracle headroom. It must do so at equal token and execution cost, and it must beat naive resampling at that same cost.
This differs from the closest prior work in specific ways. SMART models strategy choice as a sequential decision problem. It uses fallback strategies found during training to teach the model a better first choice, with the explicit aim of removing test-time refinement, and it was tested only on math. We keep the test-time loop and condition every choice on runtime feedback. RoutingGen routes coding problems into a simple or a complex reasoning path before generation, with no execution in the decision. Our routing happens after each failure and chooses among more than two moves. PairCoder comes closest. A planning agent proposes diverse algorithmic plans, and an implementing agent repairs by failure category, switching plans after repeated failure. But its switch rule is heuristic, and its within-plan repair is a fixed template. We make the whole move choice, including "switch plans now," learned and feedback-conditioned. We also measure whether learning adds anything over PairCoder's rule.
Research Program
We first define a repertoire of five repair moves that differ in reasoning style, not just wording. A direct patch edits the code with no intermediate reasoning, which addresses the "overthinking trap" documented by RoutingGen. A trace-and-fix move asks the model to simulate the failing input step by step before editing; LiveCodeBench treats execution prediction as a capability separate from generation (LiveCodeBench). A re-specification move restates input/output constraints and edge cases before rewriting, following the specification stage of Intention Chain-of-Thought. A replan move discards the code and writes a new natural-language plan under an instruction to contrast with the failed idea, borrowing the adversarial-diversity prompt from PlanSearch. A property-check move derives invariants and a minimal counterexample before editing, reusing the Tester design of PGS. PGS showed that verification accuracy (76.5%) far exceeds generation accuracy (32.4%) on LiveCodeBench Hard. It also showed that minimal counterexamples beat raw traces. We therefore feed every move the same minimized feedback, so the moves differ only in reasoning strategy.
We then collect counterfactual data. We sample initial attempts on training problems, drawn from CodeContests training data and LiveCodeBench problems released before a cutoff. At each failing state, we branch into all five moves, each with a fixed number of samples. We record whether the move reaches a passing program within that round, the change in the fraction of tests passed, and the tokens spent. This yields a contextual-bandit dataset: each failure state comes with measured outcomes for every action.
On this data we train a selector $\pi(a \mid s)$, where $s$ is the failure state and $a$ one of the five moves. It picks
$$a^* = \arg\max_a \; \hat{p}(\text{success} \mid s, a) - \lambda\, \hat{c}(s, a),$$
where $\hat{p}$ is the predicted chance that move $a$ fixes the program this round, $\hat{c}$ its predicted token cost, and $\lambda$ a price per token that we sweep to trace a cost–accuracy curve.
The selector's input comes in two parts. One is hand-built feedback features: error type (runtime error, wrong answer, time limit), pass fraction, whether the last edit changed the failure signature, and round index. The other is a small text embedding of the problem and the minimized counterexample. The selector is deliberately external and small. Chain of Methodologies and SELF-DISCOVER both report that 7B-scale models fail at choosing their own methodology, looping or defaulting to the first option. An external selector sidesteps that failure and lets us ask whether small coding models benefit more than large ones.
The analysis side matters as much as the method. We will publish the oracle-headroom measurement and the confusion structure of which move wins under which feedback. We will also report how the winning move shifts with round index, which tests whether escalation from patching to replanning is learnable or better done by a fixed schedule.
Baselines
The naive reference is standard self-debugging. The same backbone receives the minimized failure and a single fixed repair prompt each round for up to $K$ rounds. CodeTransOcean documents the plateau of this approach, from 48.57% to 52.57% debugging success over three rounds of compiler feedback on its translation subset. The "do nothing" point is the single-attempt pass rate. For the LiveCodeBench self-repair setting, the benchmark paper reports GPT-4-Turbo rising from 24.5% to 36.9% when given feedback (LiveCodeBench). This anchors how much a single fixed repair round buys.
The best single fixed move is chosen on validation data, for example always trace-and-fix. It isolates whether any selection is needed at all. A fixed escalation schedule (patch, then trace, then replan) tests whether a simple hand-written order captures the benefit.
The strongest structural competitor is PairCoder's switch-on-repeated-failure rule, reimplemented on our backbones with our moves. It reports CodeContests pass@1 rising from 6.06% to 15.15% with GPT-3.5-Turbo (PairCoder). PGS is the strongest fixed-template refiner. It reports a HumanEval bug-fix rate up to 67.9%, and it is literally our property-check move used every round.
The ablation that isolates our key ingredient is a selector trained on the same data but given only problem features and round index, with no execution feedback. This is the RoutingGen/SMART-style upfront router extended to multiple rounds. If it matches the full selector, feedback conditioning is not what matters.
Every method runs on the same backbones: Qwen2.5-Coder-7B, DeepSeek-Coder-6.7B, and one strong API model. Every method uses the same minimized feedback, the same held-out problems, and the same prompt-tuning budget on validation data. Every method is plotted against cumulative generated tokens and number of executions, not rounds. Beating a baseline means a higher pass rate at every matched budget across the sweep, with a bootstrap confidence interval over problems that excludes zero. It also means recovering at least a third of the gap between the best fixed move and the per-round oracle. Below that, the added machinery is not worth it.
Why not just…?
Why not just resample from scratch with the same budget? This is the most serious threat. Restarts escape a flawed initial idea for free, and much of the apparent value of self-repair can disappear once resampling is matched on tokens. PlanSearch shows plain resampling suffers diversity collapse: Claude 3.5 Sonnet reached 55.6% on LiveCodeBench with repeated sampling against 77.0% with plan-level search (PlanSearch). That evidence concerns resampling versus planned search, not resampling versus repair. We therefore include token-matched best-of-N resampling as a baseline curve, and "replan" as one of the selector's actions. If the selector learns to always replan and matches resampling, we will report that repair adds little on these benchmarks.
Why not just let the model choose its own strategy each round, as Chain of Methodologies does? Self-selection costs an extra call per round. It is reported to collapse into loops at 7B scale (Chain of Methodologies). It has also never been evaluated on code repair. We run it as a baseline with a justify-then-act prompt. If it matches the learned selector on the large backbone but not the small ones, that is itself a useful result: the external selector would then matter mainly for small models.
Why not just train the repair policy end to end with reinforcement learning, as TGPR does? From Search to Policy moves search into training and reports 41.2 on LiveCodeBench with a 7B model under greedy decoding, above GPT-4o-mini's 36.7. That approach is complementary rather than competing. It improves how well a single repair is written, while we choose which kind of repair to attempt. The selector can sit on top of a TGPR-style model, and we will test that combination if a trained checkpoint of that kind is available to us. If it is not, we will say so rather than claim coverage.
Proposed Experiments
The first experiment is the gating one. On held-out LiveCodeBench problems released after the training cutoff, and on CodeContests test problems, we compute the per-round oracle that picks the best move with hindsight, and compare it to the best fixed move at equal cost. If the oracle gain is small, the variance in best strategy across states is too small to learn from. We would then stop and publish the negative result, which would say that the refinement plateau is not caused by uniform prompting.
If the headroom is real, the main experiment compares the full selector against all baselines above as pass-rate-versus-token curves. It uses the three backbones, LiveCodeBench, CodeContests, and HumanEval+/MBPP+ as a near-ceiling sanity check. It also uses the CodeTransOcean translation subset, where we test whether the selector learns to avoid step-by-step reasoning for the local, line-level fixes that the benchmark authors found chain-of-thought damages.
The decisive checks are three. The first is the feedback-free selector ablation. The second is the token-matched resampling curve. The third is an extension of the round budget to $K=8$, testing whether switching moves keeps gains coming past the one-to-three-round plateau where fixed loops stall.
Repository-level repair on SWE-bench is out of scope for this study. Its cost and localization problems would confound the strategy question, and we would only attempt it if the function-level results are clearly positive.
References
- SMART: Self-Learning Meta-strategy Agent for Reasoning Tasks
- Intention Chain-of-Thought Prompting with Dynamic Routing for Code Generation
- A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Planning in Natural Language Improves LLM Search for Code Generation
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- Chain of Methodologies: Scaling Test Time Computation without Training
- SELF-DISCOVER: Large Language Models Self-Compose Reasoning Structures
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement