Failure Localization for Iterative Code Repair

When a language model writes code that fails its tests, the usual fix is to show it the error and let it try again. This loop helps, but its gains flatten after a few rounds, it multiplies the time and tokens spent per problem, and on public benchmarks part of its apparent success may be recall of memorized solutions rather than real debugging. We propose that much of this waste comes from one conflation. A failing program can be wrong for two different reasons. The model may have misread the problem, which is a perception error. Or it may have understood the problem and implemented it badly, which is a reasoning error. Patching code cannot fix a misreading, so rounds spent patching in that case are wasted.

The proposal splits code generation into two stages. In the first, the model turns the problem story into a compact, explicit specification: input format, output format, and a deliberately naive brute-force reference program. In the second, it writes the real solution against that specification. Each failure is then routed to the stage that caused it. If the brute-force reference itself fails the given examples, the model re-reads the problem. If the reference passes but the fast solution disagrees with it, the model receives one small concrete counterexample input and patches its code. This gives a cheap, black-box way to spend refinement effort where it can help. It also gives a natural stopping rule and a representation that should be less sensitive to benchmark contamination, because the reasoning stage sees an abstract specification rather than the familiar surface text of the problem.

Core Claim

Routing each test failure to either the problem-reading stage or the implementation stage should reach the accuracy of multi-round raw-feedback self-repair with fewer rounds and tokens. The gap between a model's score on original benchmark problems and on re-skinned or re-symbolized variants should also shrink. The mechanism is not the brute-force reference alone, since differential testing against a slow solver is an old idea. It is the decision the reference makes possible: a failure is classified as a misreading or a bug, and the two receive different kinds of feedback. If routing adds nothing over differential testing alone, the decoupling claim is false. We treat that as a live outcome.

Where This Sits

Three lines of work motivate the split. Neuro-symbolic systems separate a neural perception front end from a symbolic reasoning back end. NePTune turns visual queries into Python over VLM-scored boxes and keeps 69.69% on Ref-GTA where the monolithic InternVL2.5 falls to 6.95%. It does no repair, though, and reports that some generated programs have logic or arity errors. PEIRCE runs a conjecture-criticism loop with symbolic provers over natural-language entailment and names the latency cost without measuring or reducing it. We carry the perception/reasoning split into code synthesis, where both stages can be executed and checked.

Execution-grounded repair supplies the feedback signal. TraceFixer shows that giving a repair model the "desired state" at the point of divergence improves over code-only repair by 13–20%. However, it assumes an oracle provides that state. Teaching LLMs Program Semantics via Symbolic Execution Traces finds that symbolic traces help a great deal but exhaust output-token budgets. Our spec-derived reference plays the role of TraceFixer's oracle without external supervision. Our feedback is a minimal counterexample input rather than a long trace, which targets the token-budget problem directly.

Two further lines constrain the design. Work on refinement limits shows that raw feedback plateaus (Yan et al.). VERINA finds that 64 compiler rounds lift proof success only to about 20% while models thrash. SlopCodeBench finds that repeated self-editing erodes code structure in 77% of trajectories. ICER finds that a relaxed acceptance rule converges in 1–2 iterations, which suggests the useful work in refinement is concentrated early. Together these argue for few, targeted rounds rather than many generic ones.

Contamination work defines the evaluation regime. LiveCodeBench collects problems released after model cutoffs and includes a single-turn self-repair scenario. DyCodeEval re-skins HumanEval and MBPP-Sanitized stories while holding the algorithm fixed. It shows a model fine-tuned on leaked HumanEval scoring roughly 80% on the static set but roughly 13% on the variants. A²RBench shows a 17.7% drop under symbolic token remapping. None of these evaluates a multi-turn repair loop on dynamic variants, and none measures repair cost.

Research Program

The system has three parts, all usable through a black-box API.

The perception stage reads the problem statement and emits a specification. The specification contains a typed input grammar, a short restatement using neutral variable names, and a brute-force reference program that favors obvious correctness over speed. The model is instructed to enumerate or simulate directly.

The reasoning stage writes the intended solution from the specification. It sees the original statement only as optional context, and we ablate that access.

The router runs whenever the candidate fails. It first checks the reference against the provided example tests.

A failing reference is labeled a perception error. The model gets back the mismatching example and is asked to revise the specification, not the code.

A passing reference means the router samples small random inputs from the grammar and runs both programs. The first disagreement is shrunk to a minimal input and returned as the only feedback. This is a reasoning error.

Refinement stops once the candidate passes the given tests and agrees with the reference on a fixed number of random small inputs, or when a round budget runs out. This is the stopping criterion that RepoCoder leaves open.

The analysis side has two goals. The first is to measure how often failures are perception versus reasoning errors on original versus re-skinned problems. We predict that re-skinning mainly increases perception errors. This is a testable reading of DyCodeEval's collapse that the paper itself does not decompose. The second is a feedback-scramble control: the same loop is run with feedback taken from a different problem. If raw self-repair still gains on static HumanEval under scrambled feedback but not on DyCodeEval variants, those gains reflect a memory cue, not debugging. Knowing this would change how refinement results on contaminated sets are read.

A known weakness is that the reference can itself be buggy or too slow even on small inputs. We will report how often the router mislabels a failure, judged against hand annotation of a sample. We will also report what fraction of problems admit a feasible brute-force reference at all.

Baselines

The naive reference point is single-pass generation with the same model and no feedback. It anchors how much any loop adds.

The strongest existing competitor is multi-round raw-feedback self-repair in the LiveCodeBench style. The model sees the failing test and the interpreter output, then rewrites. We extend it from LiveCodeBench's single turn to the same round budget our method gets. It is the strongest competitor because it is what practitioners actually deploy. The anchor number from the file is GPT-4-Turbo on LiveCodeBench, which goes from 24.5% to 36.9% Pass@1 with one round of error feedback (LiveCodeBench). On HumanEval, OpenCodeInterpreter-33B goes from 79.3% to 82.9% with execution feedback (OpenCodeInterpreter). For either on DyCodeEval variants, or for the token or wall-clock cost of self-repair, we found no reported result on Lacuna, so we measure those ourselves.

The key ablation is differential testing without routing. The same brute-force reference and counterexample machinery are used, but every failure is sent to the implementation stage and the specification is never revised. A second ablation removes the counterexample and returns raw tracebacks inside our two-stage pipeline. It separates the value of compact feedback from the value of routing.

Fairness rules apply to all arms. All methods use the same models: one open-weight coder from the Qwen2.5-Coder or DeepSeek-Coder families, and one API model. All use the same problems, the same prompt-tuning budget of a small number of prompt variants chosen on a held-out split, and the same sandbox. Comparisons are drawn as accuracy against total generated tokens and against wall-clock time, so our up-front specification cost is charged in full.

Our method beats the strongest baseline only if two things hold. It must match or exceed raw self-repair accuracy at equal or lower token cost on post-cutoff LiveCodeBench problems. It must also shrink the static-to-variant gap on DyCodeEval. Beating it on static HumanEval alone would not count.

Why not just…?

Why not just run more rounds of raw self-repair? More rounds is the cheapest thing to try, and it is the main threat on implementation bugs. The evidence says it plateaus. Gains flatten after a few iterations (Yan et al.), VERINA models thrash across 64 rounds, and SlopCodeBench shows code degrading under repeated self-editing. Our prediction is narrower: raw repair does well on reasoning errors and poorly on perception errors. The per-error-type breakdown under matched round budgets will show whether that holds. If raw repair matches us on both error types, the proposal fails.

Why not just generate a brute-force solution and test against it, without any routing? This is the most serious alternative, because most of the benefit could come from the counterexample alone. It is run directly as our key ablation. If it matches the full system, we will report that the decoupling adds nothing and that differential testing is the whole story. We expect routing to matter mainly on re-skinned problems. There, a misread reference agrees with a misread solution, so differential testing alone passes silently and only the example-test check on the reference catches the error.

Why not just use a long-reasoning model in a single pass? Test-time reasoning avoids loop overhead entirely. However, VERINA's o3 reaches only 4.9% on proof generation without verification, and the symbolic-trace study finds reasoning tokens alone exhaust budgets without grounding. We include a single-pass reasoning-mode arm, matched on tokens, on LiveCodeBench and DyCodeEval. If it dominates the accuracy–cost curve, the case for any external loop weakens accordingly.

Proposed Experiments

Main comparison. We run all arms on post-cutoff LiveCodeBench code-generation problems and on DyCodeEval variants of HumanEval (164 seeds) and MBPP-Sanitized (427 seeds). We report Pass@1 after $k$ rounds for $k$ up to 5. We also report total generated tokens, number of model calls, and sandbox-inclusive wall-clock time. On DyCodeEval we additionally report the static-minus-variant gap per method.

Error decomposition. On the same problems we record the router's perception/reasoning labels for original and re-skinned versions. We hand-check 200 failures to estimate how often the router mislabels.

Remapping check. We run A²RBench items with and without symbolic token remapping. The rule-inference step counts as perception and the transformation program as reasoning. We ask whether the 17.7% remapping drop concentrates in perception errors. The top model's reported 40.9% overall accuracy serves as context, not as a target.

Scramble control. We repeat raw self-repair with scrambled feedback on HumanEval and on its DyCodeEval variants.

Decisive outcomes. The central idea is supported if three results hold together. First, routing beats the no-routing ablation specifically on re-skinned problems. Second, our method matches raw self-repair accuracy at lower token cost on LiveCodeBench. Third, the scramble control shows static-benchmark refinement gains that vanish on variants. It is refuted if the no-routing ablation ties the full system everywhere.