When a language model repairs its own code, it reads feedback from running that code: a pass/fail bit, an error message, a failing test, or a full record of how every variable changed. The field has mostly asked whether to include such feedback. This proposal asks a different question: how much information each kind of bug needs, and which information it is. Work on lossy compression in vision and text has shown that tasks degrade unevenly when information is discarded. Image classification survives heavy JPEG compression while segmentation, which needs exact boundaries, collapses. Reading text in images (OCR) loses over 30% accuracy under moderate visual token pruning, while coarse instruction-following does not. We expect code repair to follow the same pattern. A syntax or reference bug behaves like classification: it needs little more than the error line. A logic bug behaves like segmentation: it needs exact program state at the exact moment the computation went wrong, and nothing else.
We propose to measure, for each bug class, a repair–budget curve: repair success as a function of how many tokens of execution feedback the model sees and which tokens they are. We then test one specific compression built from that measurement. It is a divergence slice: the variable values that differ between a failing run and the nearest passing run, kept only in a short window around the first step where the two runs disagree. The bet is that two known failures are compression artifacts rather than fundamental limits. The first is the DebugBench "feedback paradox," where tracebacks hurt on logic bugs. The second is the finding that self-repair barely beats independent resampling at equal compute. Raw tracebacks and full traces both sit on the wrong part of the logic-bug curve: too long, and yet missing the state that matters.
Core Claim
Our claim is that repair success for logic bugs has a cliff. Below a certain amount of correctly placed state information, extra feedback does not help, and noisy feedback actively hurts. For syntax and reference bugs, the curve is flat or declining beyond an error line. If this is right, a single fixed compression rule cannot be best for both classes. The rules in question include PGS's shortest counterexample, the "Concise" trace format, and StepCodeReasoner's 10-line cap. More importantly, which tokens are kept should matter more than how many. A divergence slice at a few hundred tokens should beat both a traceback and a full trace on logic bugs, and should beat a random or position-shuffled slice of identical length.
To make the measured object explicit, let $R_c(b, s)$ be the fraction of bugs of class $c$ repaired within a fixed number of rounds. Here $b$ is the feedback budget in tokens per round and $s$ is the selection rule that decides which trace content fills that budget. The empirical contribution is estimating $R_c$ across classes and selection rules. The method contribution is a selection rule $s$ that reaches the logic-bug plateau at small $b$.
Relation to Prior Work
Several papers motivate this proposal by showing that feedback form matters, though each uses a fixed rule.
- DebugBench builds 4,253 debugging instances with labeled bug types. It finds that tracebacks help syntax and reference bugs but hinder logic bugs. GPT-4 reaches 75% overall and falls to 55.9% on four-bug instances. We treat this paradox as the central phenomenon to explain.
- Do Code Semantics Help? compares five trace formats. It finds that a Concise format, which drops unchanged variables, wins at inference time in 65 of 70 settings, and that traces add up to 10.85% for GPT-4o on MBPP-R. Its compression is one static filter, not varied by bug type or budget.
- PGS selects the failing counterexample with the fewest tokens and checks properties rather than raw tests. It reaches up to 67.9% bug fix rate on HumanEval. It assumes minimal is always best, which is exactly what our curves test.
- StepCodeReasoner caps traces at 10 lines with no print anchors inside loops. That cap is a hand-set point on an unmeasured curve, and the paper targets reasoning and generation rather than multi-round repair.
One paper supplies the key ingredient but under an unrealistic assumption. TraceFixer shows that the last few traced lines plus the "desired state" at the divergence point improve repair by 13–20% over code alone. It requires an oracle for the desired state and works with a small encoder-decoder on single-line fixes. Our divergence slice replaces the oracle with the trace of a nearby passing input. That input is found automatically from passing tests or by shrinking and mutating the failing input, in the spirit of validity-preserving delta debugging.
Other papers constrain the design.
- Demystifying GPT Self-Repair shows that at matched compute, self-repair gains only 5–8% over resampling on APPS for GPT-4 and loses for weaker models. GPT-4 fixes 52.6% of programs with human feedback versus 33.3% with its own. This forces every comparison we make to be compute-matched against resampling. It also tells us the bottleneck is diagnosis, which better feedback could in principle address.
- Can visual input be compressed? and Redundancy, Isotropy, and Intrinsic Dimensionality find that random pruning and naive truncation are strong baselines for coarse tasks. So we must show that selection matters for code. Otherwise the sensible conclusion is "just truncate."
- The vision results on JPEG defects and multi-task split computing supply the hypothesis that localized tasks have cliffs. We only borrow the hypothesis, not a method.
- Two papers on repository context and agent memory are relevant. COCOGEN starts from minimal repository context and expands it on compiler "information requests." ACON learns natural-language compression guidelines for agent histories from contrastive failures, cutting tokens by 26–54%. Both suggest strong alternatives that we run as baselines rather than dismiss: letting the model request detail itself, and learned summarization.
Research Program
The first component is a trace-compression ladder for Python. We instrument programs to record line-level variable states. We then render feedback at increasing fidelity:
- a pass/fail bit;
- the error message or traceback;
- the minimal failing input with expected versus actual output, PGS-style;
- the divergence slice;
- the full Concise trace;
- the raw full trace.
The divergence slice is the new rung. We align the failing run with the nearest passing run by line number and loop iteration. We locate the first step where any shared variable differs. We then emit only the differing variables, plus their values in the passing run, over a window of $k$ steps before and after that point. Varying $k$ and the number of variables yields a continuous token axis for this selection rule. Truncation, random pruning, and length-matched summaries yield matched points on the same axis for competing rules.
The second component is the measurement itself. We use the Python3 portion of DebugBench, which already labels syntax and reference, logic, and multi-bug instances, so the class variable is given rather than inferred. We complement it with MBPP-R from the trace study, HumanEval and LiveCodeBench self-repair problems with the model's own failed attempts, and CodeNet4Repair from RePair for real human bugs. For each class, rule, and budget, we run multi-round repair with three rounds, matching the observed plateau after two to three rounds in iterative refinement. We record repair success, tokens consumed, and executions. Models include open-weight code models in the 7B–34B range, which is the regime where Olausson et al. found repair losing to resampling, plus one strong API model.
The third component is a deployable policy. It needs no bug classifier: the cheapest rung is always tried first and fidelity is escalated on failure. Class information enters only through the shape of the measured curves, which fixes the order and the budget of each rung. We deliberately avoid a learned "predict the bug type, then pick fidelity" router. A skeptic would rightly say that plain escalation does as well, and we test that directly.
Baselines
The naive reference point is repair with only the pass/fail bit and the failing assertion. Next to it sits independent resampling at temperature 0.8 with the same total token budget and the same number of test executions. Following Demystifying GPT Self-Repair, we plot pass rate against tokens spent rather than rounds. Beating resampling means a higher pass rate at every budget on the logic-bug subset for at least the open-weight models where that paper found repair losing. No reported result found on Lacuna for resampling on DebugBench, so we establish this reference ourselves.
The strongest existing feedback method is the Concise trace from Do Code Semantics Help?. It reports up to a 10.85% improvement for GPT-4o on MBPP-R from inference-time traces, and its format won in 65 of 70 settings. We run it with identical models, problems, rounds, and prompts. We also run it truncated to the divergence slice's token count, so we compare both at its natural length and at matched length. No reported result found on Lacuna for Concise traces on DebugBench. Beating it means higher logic-bug repair at fewer tokens and no loss on syntax bugs.
The second strong competitor is PGS-style minimal counterexample feedback. It reports up to 67.9% bug fix rate on HumanEval in PGS. We reimplement only its minimal-input selection, with oracle tests rather than LLM-generated properties, so the comparison isolates feedback form. We also run it with the full trace of that minimal input, because shrinking the input already shrinks the trace.
Two learned or agentic baselines get the same tuning budget we spend choosing $k$. The first is an ACON-style summarizer whose guidelines are optimized on a held-out split by contrasting failures under compression, as in ACON. The second is a COCOGEN-style on-demand variant: the model starts with the traceback and may call a tool to request variable values at a line of its choosing, following COCOGEN.
The key ablations isolate placement. One applies random pruning of the full trace to the slice's length. The other is a shuffled slice: the same window size and variable count, centered on a random step instead of the true divergence. If either matches the real slice, the proposal's central ingredient fails.
Why not just…?
Why not just always use Concise traces? This is the most likely way the proposal turns out unnecessary. If Concise traces already sit on the logic-bug plateau, and do not hurt syntax bugs relative to tracebacks, then the bug-class story reduces to "use Concise." The evidence that they may not suffice is indirect. The format was evaluated on MBPP-R and generation tasks, not on DebugBench's labeled logic and multi-bug instances where the paradox appears. It also keeps every changed variable across all steps, which for loops is still long. The class-by-rule interaction in our curves settles this. A flat interaction is a reportable negative result, and we would recommend Concise.
Why not just resample more? Olausson et al. show this wins for weaker models at matched compute, so it is a real threat, not a strawman. Their repair feedback was error text and self-critique, and their human-feedback result suggests the limit is diagnosis quality. Our compute-matched curves on logic bugs test whether better-placed state closes that diagnosis gap. If resampling still wins for 7B–13B models, we report that the gain from compression is confined to stronger models.
Why not just let the model ask for what it needs? COCOGEN shows that model-driven context requests work for missing symbols. The same idea could let a model pick its own trace window. The open question is whether models know where to look on logic bugs. Olausson et al.'s self-feedback numbers suggest they often do not. The on-demand baseline measures this directly. If it matches the divergence slice at equal tokens, the contribution narrows to the measurement, and we would say so.
Why not just shrink the failing input? A smaller failing input gives a shorter trace for free, and PGS shows minimal inputs help. We include "minimal input plus full trace" as a baseline for this reason. The divergence slice should still help when the bug manifests deep in a computation even on small inputs, such as accumulator or off-by-one errors in loops. That is where we expect the gap.
Proposed Experiments
The first decisive check is whether curves differ by class. We fit repair success against log feedback tokens, separately per DebugBench class and selection rule. The hypothesis requires a significant class-by-budget interaction: logic-bug curves rise steeply over a narrow band of divergence-slice budgets, while syntax curves are flat beyond the traceback.
The second check is whether the paradox dissolves. DebugBench's finding is that tracebacks fall below minimal feedback on logic bugs. The divergence slice must rise above both on logic bugs while costing fewer tokens than the full Concise trace.
The third check is placement. At identical token counts, the true slice must beat the random-pruned trace and the shuffled slice on logic bugs. If it does not, the compression-sensitivity framing is wrong for code, as it was for coarse visual tasks, and only the amount of information matters.
The fourth check is the compute-matched comparison with resampling on DebugBench logic and multi-bug items and on LiveCodeBench self-repair. LiveCodeBench reports GPT-4-Turbo rising from 24.5% to 36.9% with self-repair in LiveCodeBench, which gives a sanity anchor for the strong model.
Finally, we check transfer. Curves estimated on DebugBench's implanted bugs are used to set the escalation schedule on CodeNet4Repair's human bugs. Success requires the schedule to beat Concise traces at matched tokens there without retuning. Without that, the curves would be an artifact of GPT-4-implanted bugs.
References
- DebugBench: Evaluating Debugging Capability of Large Language Models
- Do Code Semantics Help? A Comprehensive Study on Execution Trace-Based Information for Code Large Language Models
- PGS: Effective LLM Code Refinement via Property-Oriented and Structurally Minimal Feedback
- StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
- TraceFixer: Execution Trace-Driven Program Repair
- Validity-Preserving Delta Debugging via Generator Trace Reduction
- Demystifying GPT Self-Repair for Code Generation
- Can visual input Be Compressed? A visual input token compression benchmark for large multimodal models
- Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings
- Analyzing and Mitigating JPEG Compression Defects in Deep Learning
- A Multi-Task Supervised Compression Model for Split Computing
- Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
- ACON: Optimizing Context Compression for Long-horizon LLM Agents
- RePair: Automated Program Repair with Process-based Feedback
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code