Repair Memory Built From a Model's Own Failures

When a language model fixes code in a loop (write, run the tests, read the error, try again), it usually gets most of its gains in the first one or two rounds and then stalls. Retrieval can help here: the system looks up similar past bugs and their fixes and shows them to the model as examples. But the past bugs we index are either human commits or synthetic mutations, such as swapped variables and flipped operators. By the second round of a refinement loop, the bug in front of the model is neither. It is the model's own leftover mistake, a residual error it has already failed to fix once. We propose to treat this as a synthetic-to-real distribution mismatch that lives inside the retrieval corpus. The fix is to build the corpus from the model's own refinement trajectories: failing attempts, the execution evidence that exposed them, and the edit that eventually passed. The loop then queries this corpus again at every round, keyed by how the program failed rather than by what the code looks like.

The claim is testable and could be false. Retrieval helps a refinement loop in proportion to how closely the corpus's errors match the errors the loop actually produces. A memory of self-generated failures should therefore beat equally sized memories of human bugs or rule-based mutations, and the advantage should appear specifically in later rounds, where plain self-repair plateaus. If the human-bug corpus helps just as much, the distribution-mismatch framing is wrong and the simpler existing methods suffice.

Why this gap exists

The bug-detection literature has already shown that the source of training errors matters a great deal. On Distribution Shift in Learning-based Bug Detectors finds that a detector trained on balanced synthetic mutations reaches only 3.4% precision on real variable-misuse bugs. Its two-phase adaptation to real, imbalanced data raises this to 64.8%. Self-Supervised Bug Detection and Repair learns to inject harder single-token bugs through an adversarial game between a bug inserter and a detector. TraceFixer trains an inverted repair model to inject plausible single-line bugs. These papers motivate our framing: they show the error source is a first-order variable. They also constrain it, because every one of them treats error distribution as a training-data problem for a static detector, never as a property of what a repair loop retrieves.

On the retrieval side, RAP-Gen retrieves past human bug-fix diffs as in-context examples. It rests on the "redundancy assumption" that new bugs resemble old ones, and it retrieves once. ReCode adds two ideas: it has the model first name the algorithm type (dynamic programming, greedy, and so on), and it embeds problem text and code separately. Its corpus is human-annotated Codeforces bugs, and it explicitly positions itself as an alternative to iteration. On RACodeBench with GPT-4o-mini, it reports 38.49% versus 34.79% for iterative self-repair. ReCode is the paper that appears to contradict us, and we take it seriously. Our reading is that it compares one-shot retrieval against iteration without retrieval. It never tests iteration with retrieval matched to the loop's own errors.

Among systems that combine retrieval and iteration, ARCS retrieves once and then repairs from raw stack traces, keeping the best candidate seen so far. COCOGEN re-retrieves during the loop, but only to resolve undefined symbols named by the compiler. That is a lookup of missing context, not of how similar failures were fixed.

RefineCoder is the closest work in spirit. It also uses a model's self-generated errors, but it bakes them into the weights through fine-tuning on critiques and never retrieves them. Finally, CodeTransOcean supplies the plateau we target: GPT-3.5 debugging success goes from 48.57% to only 52.57% after three rounds.

Core Claim and Mechanism

We call a failure signature the compact execution evidence of a wrong program. Its main part is the failing test input, the exception type and location, or the first expected-versus-actual output mismatch. It also includes the algorithm label that ReCode's classifier assigns.

The memory is harvested offline. We run the backbone model's own refinement loop on a training pool of problems with tests. Every time an attempt fails and a later attempt passes, we store four things: the failing program, its signature, the diff to the passing program, and the round index. This harvesting step is the only place the loop's own residual errors are sampled. It turns the synthetic-versus-real question into a sampling choice: we draw errors from the distribution the deployed loop will actually visit, not from mutation rules.

At test time, after each failed execution, the loop builds a query from the current signature and code. It uses separate embeddings for the two, following ReCode's lesson about text and code interfering in a single embedding. It then retrieves the top few stored repairs and shows them to the model.

The imbalance lesson from the bug-detection work carries over as a gate on retrieval. Most retrieved neighbors will be irrelevant, and irrelevant examples actively mislead, echoing the distraction effect in SWE-bench. The loop therefore retrieves only when signature similarity clears a threshold. That threshold is tuned on a validation split for precision, not recall.

Research Program

The work has three parts. The first builds four corpora of equal size from the same training pool. The first corpus holds self-sourced trajectories. The second holds human buggy-fixed pairs, using the Codeforces-style data that underlies ReCode. The third holds rule-based mutations of passing solutions, using BUGLAB-style operators, repaired by reversal. The fourth holds another model's trajectories, drawn from a different backbone.

The second part runs the same refinement loop with each corpus swapped in and everything else fixed. This lets us estimate how retrieval gain depends on the match between corpus and query errors. We measure distribution match directly in two ways: the distance between embeddings of corpus signatures and test-time signatures, and the histogram of error categories, which covers wrong answer, timeout, runtime exception, and off-by-one on boundary inputs.

The third part is a round-by-round analysis. We ask whether each corpus's gain comes from the first attempt or from rounds two and beyond. The hypothesis predicts that only the self-sourced corpus keeps gains alive in later rounds.

Harvesting uses a problem pool disjoint from all evaluation sets. We deduplicate by problem identifier and by near-duplicate statement text, so that no stored repair solves an evaluation problem verbatim. This matters especially because the out-of-distribution evaluation includes AtCoder problems, and AtCoder-style problems could otherwise leak into a harvesting pool.

Baselines

The naive reference point is iterative self-repair with no retrieval. The model sees its failing code and the raw error and tries again, as in ARCS's repair loop without upfront retrieval. We run it with the same backbone, the same maximum number of model calls, and best-so-far selection. ReCode reports 34.79% for this setting on RACodeBench with GPT-4o-mini. Beating it by a clear margin is a floor, not a result.

The strongest existing method is ReCode itself, which reports a 38.49% pass rate on RACodeBench with GPT-4o-mini. On the AtCoder out-of-distribution set, it reaches 35% with 4 model calls, compared with 15 for self-repair. We rerun ReCode with GPT-4o-mini on RACodeBench, using its algorithm classifier and separate encoders. We then report accuracy against the number of model calls for both methods. Beating ReCode means exceeding its pass rate at an equal or lower call count, with the gap larger than the spread across three sampling seeds.

The baseline that isolates our key ingredient is the same loop with the same retriever, gate, and call budget, but backed by the equal-size human bug-fix corpus. This is effectively RAP-Gen-style retrieval re-issued every round. The rule-mutation corpus serves as a second control. If the self-sourced memory does not beat the human corpus, the central claim fails.

A second ablation keeps the self-sourced corpus but keys retrieval on code similarity alone, dropping the failure signature. This tests whether indexing by how a program failed is what carries the benefit.

The last baseline stores the same harvested trajectories in the weights instead of the index. We fine-tune an open 7B coder on them in the RefineCoder style, with a matched harvesting budget. We compare it against the frozen 7B model with retrieval. RefineCoder reports 83.3 pass@1 on HumanEval+ for its 7B model after three iterations, but that result is on a different benchmark and cannot be compared directly. The fair comparison is ours, run on identical trajectories.

Why not just…?

Why not just fine-tune on the self-generated repairs instead of retrieving them? This is the most serious threat. RefineCoder shows that self-generated errors make useful training signal. Retrieval has two practical advantages: it works with frozen API models, and the memory can be updated as the model or codebase changes without retraining. Neither advantage matters if fine-tuning simply wins on accuracy. The matched-trajectory comparison above settles this directly. We also test the combination, a fine-tuned model with retrieval. If fine-tuning alone dominates at equal data, we will report that retrieval's role reduces to deployment convenience.

Why not just sample many candidates and keep one that passes? Best-of-N sampling is simple and parallel. ReCode's call counts suggest that blind resampling is expensive for the accuracy it buys. We give best-of-N exactly the same number of model calls and plot both methods on one accuracy-versus-calls curve. If the curves cross, we report where they cross.

Why not just use ReCode's human-bug retrieval at every round? This baseline asks whether the redundancy assumption already covers a model's residual errors. It is precisely our human-corpus control. Only the round-by-round breakdown can distinguish the two explanations. If human bugs help as much as self-sourced ones in rounds two and beyond, the mismatch hypothesis is refuted.

Why not let the model write its own search query from the traceback, as agentic systems do? COCOGEN shows that diagnostic-driven lookups work for missing symbols. We include a variant in which the model writes a free-text query from the error and searches the self-sourced corpus. This separates the value of the corpus from the value of our signature key.

Proposed Experiments

The primary evaluation is RACodeBench, with the six out-of-distribution competitive-programming sets from ReCode as a transfer test. The metric is pass rate at matched call budgets from 1 to 8 calls, using GPT-4o-mini and an open 7B coder.

The second regime is the LLMTrans subset of CodeTransOcean, where the metric is debugging success rate after K rounds. The question there is whether the three-round plateau flattens less under self-sourced retrieval. As a secondary check on code quality, we track verbosity growth across rounds, in the spirit of SlopCodeBench. This tests whether passing fixes from memory accumulate less bloat than unaided self-repair.

Three decisive checks define success. First, the corpus-swap matrix should show self-sourced memory above human, cross-model, and mutation memories, with gain rising as signature-distribution distance falls. Second, the self-sourced advantage should concentrate in rounds two and beyond. Third, using another model's trajectories should give less benefit than using a model's own, which would confirm that the match is to a specific model's error distribution, not merely to "LLM-style" bugs. Failing the first check refutes the proposal. Passing it but failing the third would mean model-generated bugs are generically better retrieval material, a weaker but still useful finding.