Code-generating language models are increasingly run in a loop. They write a program, run it against tests, read the errors, and try again. Whether this loop beats simply drawing more fresh attempts is still disputed. Under equal compute, GPT-4's self-repair beat independent sampling on APPS by only about 5–8%, and for GPT-3.5 and Code Llama it was worse (Demystifying GPT Self-Repair for Code Generation, Is Self-Repair a Silver Bullet for Code Generation?). Yet other evaluations report large repair gains. We think one plain distinction explains much of the disagreement. A failing program is wrong either in its plan (the algorithm or approach is unworkable) or in its details (an off-by-one error, a missed edge case, a wrong type). Feedback-driven repair is good at fixing details and bad at replacing plans. Starting over is the reverse.
The second half of the argument concerns benchmark contamination, meaning test problems that leaked into a model's training data. On leaked problems, a model usually recalls the right plan, so its failures are detail-level and repair looks strong. On genuinely new problems, plan-level failures should dominate, so a repair loop that only patches details stalls. We propose to measure this split on contamination-controlled benchmarks. We then build a repair loop that first makes a cheap coarse judgment of whether the plan is sound and spends compute accordingly: it repairs when the plan looks sound and resamples a new plan when it does not. Only after that judgment does it apply fine-grained, counterexample-driven fixes.
Core Claim
The claim has three linked parts. First, the measured benefit of iterative repair over equal-compute resampling depends on contamination. Repair gains should shrink or reverse on problems released after a model's training cutoff, and the shrinkage should be explained by a rise in the share of plan-level failures. Second, a repair loop gated by a coarse plan-versus-detail decision should beat both pure repair and pure resampling on post-cutoff problems at matched compute. Third, part of the measured advantage of large general-purpose models over fine-tuned open code models on fresh problems is a plan-level advantage, not a detail-level one.
The gate borrows its logic from coarse-to-fine classification. Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification multiplies a fine classifier's probability by a separate coarse classifier's probability for the parent class, and this cuts severe mistakes. We apply the same idea to the repair decision. With $e$ the evidence available about a failing program $y$, we repair if and only if
$$ P(\text{plan sound} \mid e)\cdot P(\text{fixed within } b \text{ edits} \mid \text{plan sound}) \;>\; P(\text{fresh sample passes}), $$
where $b$ is the remaining repair budget. All three terms are estimated on a held-out calibration window. The coarse term plays the role of the parent classifier: it vetoes fine-level patching of programs whose plan is already wrong.
Relation to Prior Work
The motivating evidence comes from two directions. The self-repair studies above show that repair often loses to resampling, and that GPT-4's own feedback was wrong in 40% of cases. They ran on HumanEval and APPS, which are documented as leaked. Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models found exact solution matches for 20.8% of MBPP in StarCoderData. It also reports StarCoderBase scoring 72% on the most-leaked tenth of MBPP versus 22% on the least-leaked tenth. LiveCodeBench shows sharp drops on post-cutoff problems, and it reports self-repair gains (GPT-4-Turbo from 24.5% to 36.9% Pass@1). It does not break those gains down by contamination status or failure type, and that breakdown is our first contribution.
Contamination findings also constrain what we can expect to see. Rethinking the effects of data contamination in Code Intelligence found that only paired input-output leakage inflates zero-shot scores. Partial leakage showed no significant effect, and task fine-tuning largely erased pre-training leakage in small models. Our hypothesis must therefore hold mainly where whole solutions leaked. We test it with a similarity-graded split rather than a simple binary flag, and we report it separately for fine-tuned and instruction-tuned models.
Three systems supply methods we reuse. PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback gives fine-level feedback through property assertions and the smallest failing input. It reports a 67.9% HumanEval bug-fix rate, and it reports verification accuracy (76.5%) far above generation accuracy (32.4%) on hard problems. We reuse its counterexample search as our fine-level step. Its properties are flat, though, and it never asks whether the plan should be abandoned. Outline, Then Details (ChainCoder) separates outline from details during generation but never runs the code. We separate them in the feedback loop instead. From Search to Policy (TGPR) trains a 7B repair policy with tree search and scalar rewards, reaching 41.2 on LiveCodeBench. It gives no plan-level signal and serves as our strongest trained competitor.
One finding cuts against parts of our framing. Beyond Generalist LLMs shows that general-purpose agents waste tokens and drift in repair loops on rigid workflows. That is consistent with our view: unguided repair is costly. It is also why we treat the general-purpose model only as an optional coarse judge, never as the patcher.
Research Program
The work has three stages.
Stage 1: label failures as plan-level or detail-level. We collect failing programs on LiveCodeBench from several models and tag each failure with an oracle label. A failure counts as detail-level if some passing program for the same problem lies within a small structural edit distance. Distance is measured on syntax trees with the Dolos-based similarity used by Quantifying Contamination, and the passing programs are pooled from all models plus oracle-feedback repairs. A failure also counts as detail-level if oracle-feedback repair succeeds within two turns. Anything else is plan-level. A few hundred labels will be checked by hand to confirm the edit-distance cutoff matches human judgment.
Stage 2: build two contamination splits. The first is temporal: for each model, LiveCodeBench problems released before versus after its stated cutoff. The second is similarity-graded and applies to open models with inspectable corpora: problems ranked by their highest syntax-tree similarity to training code. On both splits we measure the share of plan-level failures and the repair-minus-resampling gain at matched compute. We also record whether "successful" repairs on leaked problems move toward known public solutions. That would mean repair succeeds by recall rather than by reasoning.
Stage 3: build and test the gated loop. The coarse evidence $e$ comes from cheap execution signals: the fraction of tests passed, timeouts on automatically scaled-up inputs (which reveal a plan whose running time grows too fast with input size), and disagreement between the failing program's approach and the majority approach among other samples. An optional fourth signal is a one-call judgment from a general-purpose model on whether the approach fits the problem's constraints. The gate is a small calibrated classifier. It is fit on the Oct 2024 to Feb 2025 window and evaluated on later problems, so calibration never touches pre-cutoff, leaked data.
Baselines
The naive reference point is single-sample generation. Above it sits equal-compute independent sampling, which draws fresh programs and keeps any that pass the visible tests. Compute is measured both in program samples and in total tokens, following the matched-budget protocol of Demystifying GPT Self-Repair for Code Generation. That paper reports GPT-4 self-repair beating this baseline by about 8% on APPS. If our gated loop cannot beat independent sampling on post-cutoff LiveCodeBench, the proposal fails.
The flat-repair baseline is LiveCodeBench's own self-repair protocol: interpreter error messages fed back to the model. LiveCodeBench reports GPT-4-Turbo rising from 24.5% to 36.9% Pass@1 under it. We rerun it at the same token budget on the same post-cutoff problems.
The strongest feedback-design competitor is PGS, with properties plus the smallest failing input, run with its published prompts and the same budget. There is no reported result on Lacuna for PGS's fix rate on LiveCodeBench; the 67.9% figure is on HumanEval. We therefore run it ourselves.
The strongest trained competitor is TGPR. It reports 41.2 on LiveCodeBench for a Qwen-7B policy versus 36.7 for GPT-4o-mini, on a temporally decontaminated window. We apply our gate on top of the TGPR policy where weights are available. Otherwise we reproduce TGPR's evaluation window, and we report on the overlap between that window and ours.
Two ablations isolate the key ingredient. The first is a gate that uses only the test-pass fraction. The second is a gate with a random decision at the same repair-versus-resample rate as ours. Beating TGPR or PGS means a statistically significant Pass@1 gain on later post-cutoff problems at equal tokens. Beating the pass-fraction gate is required before we claim any value for the richer coarse signals.
Why not just…?
Why not just draw more independent samples? For weak models on hard, fresh problems, this may be the right answer most of the time, and the self-repair studies support it. The gate falls back to resampling whenever it judges the plan unsound, so it can only beat pure resampling by repairing the detail-level failures. Stage 1 measures how common those are. If they are rare on post-cutoff problems, we will report that resampling dominates there. That is still a useful correction to repair results reported on leaked benchmarks.
Why not just use PGS's property feedback, which already beats tracebacks? PGS improves the fine-level step, but it keeps patching a program whose algorithm is wrong. Our prediction is that PGS's advantage holds on detail-level failures and disappears on plan-level ones. Breaking down PGS's fix rate by our oracle labels tests this directly. If PGS also fixes plan-level failures well, the gate adds little and we will say so.
Why not just gate on the fraction of tests passed? This is the most serious threat, and it is why the pass-fraction gate is a named ablation. We expect it to fail in a specific way. Brute-force or greedy plans often pass small visible tests and then fail on scale or adversarial cases, so pass fraction overrates them. Scaled-input timeouts and disagreement with other samples should catch these. If the pass-fraction gate matches the full gate, the contribution reduces to the gating decision itself plus the contamination analysis, and the note's claim narrows accordingly.
Why not just decontaminate the static benchmarks? Quantifying Contamination shows that string-level filtering missed exact matches for a fifth of MBPP. Temporal splits avoid relying on such filters. We use filtering only as a secondary similarity-graded split.
Proposed Experiments
The decisive analysis is an interaction test. It regresses the repair-minus-resampling gain on contamination status, with failure type (plan or detail) as a mediator. The analysis covers DeepSeek-Coder-33B, Qwen2.5-Coder, StarCoder-family models, and GPT-4o and Claude 3.5 Sonnet as general-purpose references.
The hypothesis is confirmed if three things hold. The gain is larger before the cutoff than after it. The plan-level share rises after the cutoff. Holding failure type fixed removes most of the contamination effect.
For the general-purpose-model claim, we decompose the post-cutoff gap between general-purpose and open fine-tuned models into plan-level and detail-level failure rates. LiveCodeBench reports DeepSeek-Coder-33B trailing GPT-4 by 16.2 points on Easy problems. We predict most of that gap is plan-level.
The gated loop is evaluated on later post-cutoff LiveCodeBench problems against all baselines at three token budgets, with BugSourceBench from Anchored Self-Play for Code Repair as a check that the gate transfers to realistic, human-written bugs. A clear negative result would also be informative. That result would be no contamination interaction, together with no gain from the gate over the pass-fraction ablation. It would mean the self-repair debate is not explained by plan-level failures and must be explained some other way.
References
- Demystifying GPT Self-Repair for Code Generation
- Is Self-Repair a Silver Bullet for Code Generation?
- Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification
- Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Rethinking the effects of data contamination in Code Intelligence
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement
- Beyond Generalist LLMs: Specialist Agentic Systems for Structured Code Workflow Execution
- Anchored Self-Play for Code Repair