When a language model repairs code by running it and reading the results, it faces a bad trade-off. Bare test failures ("expected 7, got 5") say too little about where the bug is. Full execution traces, which are line-by-line logs of every variable change, say too much, and models get worse as those logs grow. We propose to split the repair loop into two steps. First, a planning step writes down a small number of testable predictions about what the program's variables should hold at named points if the code were correct. Then the program is run with checks inserted only at those points, and the model that writes the patch sees only the predictions that turned out false. The plan is the filter: the model decides what to look at before it sees any runtime output, so the trace never has to be compressed afterwards.
This combines three findings that have so far stayed separate. Verbose runtime logs hurt repair. Supplying correct "desired states" at the point of failure helps repair, but only when an oracle provides them. Separating high-level planning from syntax-heavy generation helps agents. Because the planner names its probe points before the patch is written, the sandbox can also be started and instrumented while the language model is still generating. We treat this hidden startup cost as a measured constraint on the design, not as its main selling point.
Core Claim
A repair loop in which a planning module generates expected intermediate states, and execution returns only the falsified expectations, will match or beat LLM-summarized traces on fix accuracy. It will use far fewer feedback tokens and no extra summarization call. It will also clearly beat both error-only feedback and raw-trace feedback. The specific new element is that the desired state is produced by the model from the specification rather than supplied by an oracle, and that it is used as a query over the execution rather than as passive context.
The closest work defines what must be different. Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs supplies the motivation. It shows a "complexity penalty," in which repair accuracy falls as trace length and variable-change counts grow. Its remedy is a second GPT-4-32k call that compresses the raw trace after the fact. On HumanEval-Java, this optimized trace reaches a Correct Fix Accuracy of 0.572 against 0.511 for error-only feedback. We replace post-hoc summarization with pre-registered probes.
TraceFixer supplies the key ingredient. It shows that adding the "desired state" at the divergence point improves a CodeT5 repair model by 13–20% over code-only inputs. However, it takes that state from a ground-truth oracle and lists automating it as an open limitation. We generate the desired state with the planner and test whether a non-oracle version keeps the benefit.
Lumos and MAGIS supply the decoupled structure. Lumos separates planning, grounding, and execution modules and adds an iterative variant that replans after each execution step. MAGIS splits GitHub issue resolution into manager, file-finding, developer, and QA roles. In both systems the planner reasons about subgoals or files, never about program memory states, and the runtime feedback returns to the model unfiltered.
A constraint we must face
The same trace paper reports that GPT-4's mental execution matches the true trace only 15–50% of the time. This could appear to doom model-generated expectations. We think it does not, for a specific reason. That probe asked the model to predict what the given code does. Our planner predicts what correct code should produce, which is closer to reading the specification than to simulating an interpreter.
Still, some expectations will be wrong, and a falsified prediction may reflect a planner mistake rather than a bug. The design handles this by showing the patch-writing model the expectation, the observed value, and the line, and leaving the judgement to it. We will also measure planner error directly by running the reference solution and checking which expectations it satisfies. If planner expectations are mostly wrong even against reference solutions, the central hypothesis fails. We state this as a falsification criterion.
Research Program
The system has three parts. The planner reads the specification, the failing test, and the buggy code. It outputs a short repair hypothesis plus $k$ probes, each a source location paired with a predicate over local variables (for example, "after line 14, lo <= hi and sum == 10"). We cap $k$ between 3 and 8. The executor rewrites the program to evaluate those predicates without halting, reruns the failing test, and returns the first falsified probe, any others that failed, and their observed values. The coder writes a patch conditioned on the hypothesis and the falsified probes. If tests still fail, the planner revises its hypothesis and probes. Following Fang et al., we keep rollback to the last non-regressing patch.
We will analyze three things. The first is planner expectation accuracy against reference solutions, broken down by problem difficulty. The second is the relationship between feedback length and fix rate, testing whether probe feedback escapes the complexity penalty. The third is behavior across iterations, since Yan et al. report that feedback loops plateau after two to four rounds with repeated surface fixes. Our hypothesis is that revised probes move the investigation to new locations instead of repeating it.
On the systems side, the probe specification is fixed before the patch exists. Two things can therefore start in the background while the coder generates. The executor can provision a sandbox and re-run the current buggy program with the new probes. The same overlap idea appears in QoS-Aware and Cost-Efficient Dynamic Resource Allocation for Serverless ML Workflows, which provisions functions in the background while active tasks run. The startup phases worth hiding are runtime and dependency initialization, which The Serverless Computing Survey identifies as dominant over container boot.
Baselines
The naive reference point is error-only iterative repair. The same model receives the failing test's assertion message and stack trace and gets the same iteration budget. On HumanEval-Java with GPT-4, the trace paper reports a Correct Fix Accuracy of 0.511 for this setting. We rerun it ourselves with our model, temperature, and attempt budget rather than relying on that figure.
The second baseline is raw-trace repair, using full line-level variable logs in the style of PySnooper, truncated to the context limit. This tests whether more information helps at all.
The strongest competitor is summarize-then-repair from Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs, at 0.572 Correct Fix Accuracy on HumanEval-Java with GPT-4. It is the strongest because it directly addresses trace noise and is the only reported method that beats error-only feedback with traces. We will reproduce it using the same base model for both summarizer and repairer, the same number of repair attempts, and the same sampling settings. We report accuracy at matched total tokens and matched API calls, because summarization adds a call that our method does not.
To beat it, we need either higher Correct Fix Accuracy at equal or lower cost, or equal accuracy with a large cut in feedback tokens and wall-clock time. A higher number obtained only by spending more calls does not count.
At repository scale on SWE-bench Lite, we compare against MAGIS, which reports 25.33%, under the same base model. Here our probe feedback replaces its QA-to-developer feedback channel.
Three ablations isolate the key ingredient. Random-location probes with the same $k$ test whether the planner's choice of location matters. Value-dump probes at planner-chosen locations, with no expected value, test whether the expectation matters or only the selection of what to show. Oracle-expectation probes, built from reference-solution states, give an upper bound in the style of TraceFixer. If value-dump probes match full expected-state probes, the desired-state idea is not doing the work and we will say so.
For latency, the baselines are cold start on every execution, a fixed keep-alive window, and a pre-warmed pool. All three are compared at equal memory budget, mirroring the comparison in Serverless in the Wild.
Why not just…?
Why not just summarize the trace with an LLM, as prior work does? This is the real threat. On small Java functions, a good summarizer may recover the same information that probes would. The decisive comparison is accuracy per unit of cost. Summarization always pays for a full trace plus an extra model call. Probes pay only for $k$ predicates.
We also expect the gap to widen with trace length. Summarizing a long trace is itself a long-context task subject to the same distraction, while probe cost does not grow with trace length. If summarize-then-repair matches probes on accuracy and its cost overhead turns out small in our measurements, the proposal reduces to an efficiency result. We would then report it as that.
Why not just slice the trace heuristically around the failing line? TraceFixer uses hard-coded partial trace slices, and this is cheap. The weakness is that the line where a bug shows up is often far from where the state first goes wrong. The random-location and failing-line-slice conditions test this directly. We will report fix rates on the subset of bugs where the reference fix lies more than a few lines from the crash site, since that is where planner-chosen probes should matter. If slicing ties probes there, the planner adds nothing.
Why not just keep sandboxes warm and ignore cold starts? For a single agent on one repository, a warm container is probably enough, and we will not claim otherwise. The overlap matters when many repair loops run concurrently under a memory budget. That setting resembles the skewed, bursty demand described in Serverless in the Wild, where fixed keep-alive wastes memory or still leaves invocations cold.
We will replay concurrent repair sessions with realistic gaps between language-model turns and measure stalled wall-clock time per fix at equal memory. If a fixed warm pool erases the difference, the systems component is unnecessary. The accuracy claim stands independently of it.
Why not just use chain-of-thought in one model? A single model could reason about expected states inline before writing the patch. We test this with a single-call variant that produces the same hypothesis and probes in one pass. The difference is that the execution check can then only happen after the patch is written, so the feedback concerns the patched program rather than the original. Lumos's reported gains, a 24.5% higher step success rate than WizardLM-30B on Mind2Web, motivate separating the modules, but that evidence comes from web tasks. Whether it transfers to repair is something this variant will show.
Proposed Experiments
We start at function level on HumanEval-Java, Refactory, and RunBugRun, using the repair settings from the trace paper. HumanEval-Java has the reported 0.511 and 0.572 Correct Fix Accuracy reference points. For Refactory and RunBugRun, no reported fix-rate number was found on Lacuna, so we establish all baselines ourselves.
We also run the TraceFixer real-world set of 20 Python bugs, on which TraceFixer fixed 10 of 20 using oracle desired states. This tests how close planner-generated expectations come to oracle ones. We then scale to SWE-bench Lite (SWE-bench), where container startup for repositories such as django is non-trivial. This is where we measure latency.
The metrics are fix accuracy, feedback tokens per iteration, model calls per fix, wall-clock time per fix split into model time and sandbox time, fraction of sandbox startup hidden behind generation, and planner expectation accuracy against reference solutions.
Three checks decide the outcome. First, on HumanEval-Java with matched base model and call budget, probe feedback must reach at least summarize-then-repair's accuracy with substantially fewer feedback tokens. Second, expected-state probes must beat value-dump probes at the same locations, which shows that model-generated desired states carry signal as TraceFixer's oracle states do. Third, fix accuracy under probe feedback must stay flat as the underlying trace length grows, while raw-trace and summarized-trace accuracy decline. Any one of these failing narrows the claim. The second failing would remove the core idea.
References
- Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs
- TraceFixer: Execution Trace-Driven Program Repair
- Lumos: Learning Agents with Unified Data, Modular Design, and Open-Source LLMs
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- MEIC: Re-thinking RTL Debug Automation using LLMs
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- QoS-Aware and Cost-Efficient Dynamic Resource Allocation for Serverless ML Workflows
- The Serverless Computing Survey: A Technical Primer for Design Architecture
- Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
- SWE-bench: Can Language Models Resolve Real-world Github Issues?