When a language model fixes code in a loop, it writes a patch, runs the tests, reads the error, and tries again. Most reported gains from these loops come from public benchmarks whose historical bugs and fixes are likely in the models' training data. So we cannot currently tell whether the loop teaches the model something it did not know, or whether the error message simply cues it to recall a fix it has already seen. We also cannot tell how the loop copes when the bug's cause sits several function calls away from the failing test, or when a fix must ripple through a chain of dependent code.
This note proposes a test-construction method that separates these factors. We take post-2024 Python repositories and plant bugs at a controlled number of "hops" in the call and import graph from the failing test. We also rewrite historical fixes so that the memorized patch becomes plausible but wrong. The loop's benefit can then be measured as a function of two things at once: dependency depth and whether the model could have memorized the answer. A second, smaller contribution is a feedback method that maps the runtime trace onto the dependency path. We compare it against the obvious alternative of simply expanding context k hops outward.
Core Claim
Our central claim concerns measurement. The improvement from iterative execution feedback can be split into two parts. The first is genuine debugging, which should survive when the memorized answer is made wrong. The second is recall triggered by feedback, which should actively pull patches back toward the memorized fix. We expect the balance between them to depend on the dependency depth of the bug.
Concretely, we predict three things. First, on logic-rewritten tasks, successive patches will drift toward the original historical fix. Second, this drift will be stronger in supervised-fine-tuned models than in RL-trained ones. Third, the genuine-debugging part of the gain will shrink sharply with depth, roughly as DyVal-style reasoning depth predicts, rather than with the number of files touched.
To quantify the drift, let $s_t$ be the token-level similarity between the round-$t$ patch and the original (now incorrect) historical fix. Let $a_t$ be the pass rate at round $t$. We call this the recall pull:
$$\Delta_{\text{pull}} = \frac{1}{T}\sum_{t=1}^{T} (s_t - s_0),$$
where $T$ is the number of feedback rounds. A positive $\Delta_{\text{pull}}$ alongside flat $a_t$ means the loop is recalling, not debugging.
What Exists and What Doesn't
Three papers motivate the gap from the contamination side. SWE-bench Goes Live! builds 1,319 post-2024 tasks. It reports OpenHands with Claude 3.7 Sonnet falling from 43.20% on static SWE-bench to 19.25% on the live set. It also reports success falling from about 48% on single-file bugs to 0% at seven or more files. That drop mixes contamination with difficulty, and "number of files" mixes dependency depth with breadth. Memorize or Generalize? supplies our perturbation technique. It flips one core logic operation in a reference solution and measures a Memorization Risk Index, finding that supervised fine-tuning nearly doubles that index on BigCodeBench while PPO does not. However, it studies only single-turn generation of isolated functions. DyVal supplies the design principle. It generates graph-structured reasoning tasks with independently controlled depth and width, shows accuracy decaying with depth, and reports a 13.6% drop for GPT-4 when the order of premises is reversed. But it never touches code or execution.
From the code-repair side, SynFix parses a repository into a graph of files, classes, and functions. It checks one-hop neighbors for coordinated edits, and its ablation drops SWE-bench Lite resolution from 52.33% to 31.00% when the graph is removed. Its traversal stops at one hop, and it is evaluated only on static SWE-bench. DynaFix feeds bytecode-level runtime traces into an iterative loop and credits that loop with a 21.9% fix-rate gain on Defects4J. It treats the trace as flat text and uses a benchmark old enough to be memorized.
Two papers constrain how we interpret iteration. CodeTransOcean sees compiler-feedback gains plateau within two to three rounds (48.57% to 52.57% over three rounds). Do Users Write More Insecure Code with AI Assistants? warns that closed loops without external checks compound errors.
No prior work in this file measures how iterative repair behaves when both depth and memorizability are controlled. That is the gap this proposal fills.
Research Program
The first artifact is a task generator built on SWE-bench-Live repositories and their Docker environments, which are already pinned to pre-bug library versions. For each repository we build a static call and import graph in the style of SynFix's RelationGraph and pick a test. We then generate two task families.
In root-cause tasks, we inject a single logic perturbation (a flipped comparison, a swapped set operation, an off-by-one) into a function exactly $k$ hops below the failing test, for $k$ from 1 to 5. Breadth stays fixed because exactly one function changes.
In propagation tasks, we change a contract at a leaf function, such as argument order, a return unit, or a raised exception type. The resulting fix must be carried upward through $k$ callers. Width is varied separately by choosing leaves with one versus several caller chains, which separates depth from file count.
A plant is accepted only if three conditions hold: the target test fails, every other test passes, and the reference fix restores the full suite.
The second family adapts code rewriting to repository repair. For SWE-bench Verified instances, we alter the specification of the historical fix by flipping one logical operation in the gold patch. We then edit the issue text and the hidden tests to match. As a result, the memorized patch is now wrong but compiles and looks plausible. Each rewritten instance is paired with its original, so we can measure accuracy drop and $\Delta_{\text{pull}}$ round by round. Every instance is also labeled with the hop distance in the graph between the failing test and the edited code, which lets the static, rewritten, and live sets be placed on one depth axis.
The third artifact is trace-routed feedback. When a test fails, we take the runtime call stack and local variable values, as DynaFix does. We match the frames to graph nodes and give the model only the hop-ordered path from the test to the deepest frame where observed values diverge from what the assertion implies. We also give it that path's static callers, so the model can reason about propagation. This is a deliberately modest method. Its purpose is to test whether structuring feedback along the dependency path matters, beyond merely gathering the right context.
The models will be an open-weight family that has both supervised-fine-tuned and RL-trained checkpoints at matched size (the Qwen-2.5 line used in the MRI study), plus one frontier API model. Every loop gets the same budget of execution rounds ($T = 5$) and the same token cap per task.
Baselines
The naive reference point is single-shot repair with no execution feedback (round 0) under the same localization. Any gain at later rounds is measured against it, and on rewritten tasks it gives the pure-memorization floor.
The strongest existing structured method is SynFix's pipeline: graph-based localization, one-hop synchrony, and test-feedback refinement. It is the strongest competitor because it already uses a dependency graph inside an iterative loop. It reports 52.33% on SWE-bench Lite and 55.8% on SWE-bench Verified. We reimplement it on the same models, environments, and round budget, because comparing against its published GPT-4o numbers would confuse model choice with method.
Agentless is the deterministic pipeline of localization, search-and-replace repair, and reproduction-test validation from Agentless, which reports 32.00% on SWE-bench Lite. It is the reference showing that loop structure, not agent dialogue, drives results.
OpenHands is the open-ended agent baseline, and the strongest system with a reported number on live tasks: 19.25% with Claude 3.7 Sonnet on SWE-bench-Live. We run it with the same round and token caps. For our depth-controlled synthetic tasks there is no reported result on Lacuna for any system, so all synthetic numbers come from our own runs.
Two ablations isolate the key ingredient. The first, k-hop synchrony, extends SynFix's neighbor check by breadth-first expansion to $k$ hops with no trace guidance, holding the token budget equal to trace-routed feedback. The second, raw-trace feedback, supplies the same runtime trace as flat text, as DynaFix does, without mapping it onto the graph.
Trace-routed feedback beats these baselines only if two conditions hold. Its round-5 resolution must exceed k-hop synchrony by a margin outside bootstrap confidence intervals on root-cause tasks with $k \ge 3$. Its recall pull on rewritten tasks must also be no higher. Winning only at $k = 1$ would not count, because there SynFix's one-hop design already applies.
Why not just…?
Why not just stratify SWE-bench-Live by number of files edited? This is cheap, and we will do it as a validation. But file count conflates depth with breadth, and DyVal shows these two degrade reasoning differently. It also conflates both with issue difficulty and with whatever contamination remains in the 2024–2025 window. The planted tasks hold breadth fixed while varying depth. If the depth curves from planted tasks and the file-count curves from SWE-bench-Live diverge, the stratification shortcut is shown to be misleading. If they match closely, the generator mainly adds control, and we will say so.
Why not just give a frontier model the whole repository or a long agent trajectory? The SWE-bench paper reports that more retrieved context hurt localization. It also reports Claude 2 at 1.96% with sparse retrieval, and SynFix's ablation shows unstructured context replacing the graph losing more than twenty points. Still, long-context models have improved since then, so this is a live threat. The OpenHands baseline under matched token caps tests it directly. If OpenHands matches trace-routed feedback at depth 4–5, the structural claim fails.
Why not just extend SynFix to k hops? This is the most serious alternative, and it may well suffice for propagation tasks, where the fix really is "update every caller." That is why k-hop synchrony is an ablation rather than a straw man. Our bet is narrower: on root-cause tasks, expanding outward k hops floods the context with distractors that grow exponentially with $k$, while the trace picks out one path. If the two tie at matched tokens, we drop the method claim and keep the measurement contribution. The measurement contribution does not depend on the method winning.
Why not just use post-cutoff tasks to rule out contamination and skip rewriting? Post-cutoff tasks remove memorization but cannot detect it, so they give no way to observe feedback pulling a model back toward a memorized patch. Paired original and rewritten instances are the only design in this file that exposes that mechanism, which is the approach the MRI paper validated for single-turn generation.
Proposed Experiments and Decisive Checks
The first experiment produces the depth-by-memorizability grid. We report resolution after rounds 0 through 5 for each $k$ on root-cause and propagation tasks. We report the same for original versus rewritten SWE-bench Verified pairs and for SWE-bench-Live stratified by gold-patch hop distance. The decisive check is whether genuine iterative gain (round 5 minus round 0 on rewritten and planted tasks) decays with $k$ at fixed width, and whether it decays more slowly with width at fixed $k$. That pattern would carry DyVal's depth result over to repair loops.
The second experiment tests the recall attractor. On rewritten pairs we track $s_t$ and $a_t$ per round for supervised-fine-tuned and RL-trained checkpoints of equal size. The prediction is falsifiable: $\Delta_{\text{pull}} > 0$ with flat or falling $a_t$ for supervised-fine-tuned checkpoints, and a significantly smaller pull for RL checkpoints. If the pull is zero everywhere, the worry that loops on static benchmarks overstate debugging ability is weakened, and that is a publishable negative result.
The third experiment compares trace-routed feedback with k-hop synchrony, raw traces, SynFix, Agentless, and OpenHands at matched rounds and tokens. It also includes a reversed-path control that presents the same path in reverse topological order. If ordering matters, as it did for GPT-4 in DyVal, the effect should appear here.
As a sanity anchor, our SynFix reimplementation should land in the neighborhood of its reported SWE-bench Lite figure when run with a comparable model. If it does not, we report the discrepancy before interpreting anything else. Taken together, these runs settle whether execution feedback repairs or recalls, and at what depth it stops doing either.
References
- SWE-bench Goes Live!
- Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- SynFix: Dependency-Aware Program Repair via RelationGraph Analysis
- DynaFix: Iterative Automated Program Repair Driven by Execution-Level Dynamic Information
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Do Users Write More Insecure Code with AI Assistants?
- Agentless: Demystifying LLM-based Software Engineering Agents