Who Should Remember? Asymmetric Memory in Developer–Reviewer Code Refinement

When language-model agents fix code in a loop, where one agent writes patches, another reviews them, and a test runner reports failures, every published system makes an unexamined choice about memory. Either every agent sees the full accumulated transcript of past attempts, or everyone sees only the latest state. Both choices fail in known ways. Full transcripts carry early mistakes forward and inflate cost. Refinement also plateaus after three to five rounds (Yan et al.), and code erodes under repeated edits (SlopCodeBench). Forgetting everything leaves agents free to flip back and forth between two broken fixes indefinitely. This proposal tests a third option. Give memory to exactly one role. The reviewer keeps a short, test-verified ledger of what has already been tried, fixed, and broken. The developer that writes the patches is memoryless: it sees only the current code and the current verified review.

The idea comes from a result in game theory. When two learners that both remember past rounds play a competitive game, their learning spirals away from equilibrium into cycles (Learning in Multi-Memory Games Triggers Complex Dynamics Diverging from Nash Equilibrium). When only one player remembers the other's last move and the other is memoryless, the cycles collapse and play converges globally (Global Behavior of Learning Dynamics in Zero-Sum Games with Memory Asymmetry). That proof covers depth-one memory in Matching Pennies under replicator dynamics, which is a toy far from language models. We therefore use it as a design hypothesis to be tested empirically, not as a guarantee. The hypothesis is that in a patch–review loop, the stabilizing party is the one that applies pressure, the reviewer. Giving it memory should reduce oscillation and over-editing more than giving memory to both agents, to neither, or to the developer.

Core Claim

In a two-role code refinement loop grounded by a test executor, the reviewer holds a small memory and the developer holds none. The reviewer's memory is a ledger of past review claims, each labeled by what executing the resulting patch actually did. This configuration will match or beat full-transcript multi-agent review on final pass rate at substantially lower token cost. It will also cut two measurable pathologies: oscillation, meaning the reintroduction of a previously fixed test failure or a return to a previously seen program, and over-editing, meaning diff growth and breakage of previously passing tests. The reversed asymmetry, where the developer remembers and the reviewer is memoryless, is predicted to do worse at equal tokens. If the prediction holds, the gain comes from who holds memory, not how much memory exists in total.

How This Sits Relative to Prior Work

Three code systems motivate the gap. AgentCoder separates a programmer, a test designer blinded to the code, and a test runner. It shows that controlling information flow matters, since blinding removes confirmation bias. Its refinement loop, however, feeds back an unpruned cumulative transcript. Adversarial Review adds a Critic that audits a Reviewer's claims under evidence-citation constraints. It reaches the strongest numbers in the file on LiveCodeBench and SWE-bench Verified, but keeps full transcripts at about 4.5 times zero-shot token cost and reports over-editing and regressions. MAGIS partitions repository context by file, but its Developer–QA loop also uses unpruned feedback. None of the three varies who remembers the refinement history.

Two works supply methods we reuse and contrast with. STARK organizes kernel-optimization attempts as a tree and hands different roles different curated sub-histories: the planner sees the best kernels found anywhere, and the coder sees sibling-branch nodes. That is role-specific memory, but every role still receives some history, it is chosen by heuristic, and the target is speed rather than correctness. We borrow its idea that history should be curated per role, and we test the extreme case in which one role receives none. Multi-Agent Debate with Memory Masking shows that removing erroneous past turns helps debating agents: accuracy on MATH rose from 71.2% to 80.2%, with about 30% fewer tokens. Its masking is symmetric, it filters by perplexity or self-judgment, and it is applied to math without any executor. Our ledger replaces perplexity with test outcomes as the filter and applies it to one role only. A symmetric, test-filtered port of this method is one of our baselines, because it is the most direct threat to the claim that asymmetry, rather than filtering, is doing the work.

The game-theory papers constrain the design as much as they motivate it. The divergence result warns that symmetric memory can be worse than none. The asymmetry proof says depth one may already suffice, so we sweep memory depth starting at a single round rather than assuming more history is better.

Research Program

We build one loop with three components. A developer model receives the task specification, the current program (plus retrieved repository context on repository tasks), and a single structured review, and emits a patch. A reviewer model receives the specification, the current program, the latest execution report, and optionally its ledger, and emits one or more review claims. Each claim has a code location, the failure it addresses, and a proposed direction for the fix. A non-model executor runs the tests and records which tests changed status after each patch. The test designer is blinded to the code, following AgentCoder, so that tests are not tuned to the developer's assumptions.

The ledger is the object under study. After each round, the executor labels every claim the developer acted on by its effect. A claim is recorded as having fixed failing tests, broken previously passing tests, or done nothing. The ledger also stores a normalized hash of the program's syntax tree (its structure with formatting stripped) and the set of tests that have ever passed. Only these execution-verified facts enter memory. Free-text arguments from past rounds never do, which is the design choice meant to block the erroneous-memory cascades described in the debate-masking work.

A reviewer with memory is instructed to do three things: refuse claims that previously caused regressions, flag any failure that reappears after being fixed, and ask for the smallest edit that addresses the current failure. The ledger holds the last $k$ rounds, and we sweep $k$ over 1, 3, and unlimited. This directly tests whether the depth-one stabilization seen in matrix games carries over to the loop.

Six memory configurations share the same models, prompts, and executor. Full transcripts for both roles is the prevailing practice. Memoryless for both roles is the stateless reference. Reviewer-only ledger is our proposal. Developer-only ledger is the reversed asymmetry. Symmetric ledger gives both roles the same verified memory. Symmetric masked transcript is a debate-masking port that uses test outcomes instead of perplexity.

Alongside the language-model experiments, we run a small simulation study. We extend the memory-asymmetry dynamics from the file to a stylized patch game in which the developer chooses among fixes and the reviewer chooses which failure to target, at memory depths 1 to 3. The purpose is only to check whether the predicted ordering of configurations survives beyond depth one in a controlled setting. It is not meant to supply guarantees for language models.

Baselines

The naive reference point is single-shot generation by the same backbone with no refinement loop. For orientation, Adversarial Review reports zero-shot pass rates of 77% on LiveCodeBench and 71.6% on SWE-bench Verified with its backbone, and AgentCoder reports zero-shot GPT-4 at 52.2% on MBPP-ET. We rerun single-shot generation with our backbone rather than copying these figures.

The strongest existing method is Adversarial Review with full transcripts. It reports a 75.2% pass rate on SWE-bench Verified and 87% on LiveCodeBench, ahead of the five-agent MARS system's 72.6% and 82%. We reimplement its Main/Reviewer/Critic protocol on our backbone because the published numbers use a different model. To isolate memory from role count, we run our ledger variants both in its three-role form and in a two-role form.

The second strongest competitor on these tasks is AgentCoder. It reports 96.3% pass@1 on HumanEval and 91.8% on MBPP-ET with GPT-4, at 56.9K tokens. Because those benchmarks are near ceiling, we use them only as a sanity check.

Fairness rules are the same for every method. All use the same backbone model, the same blinded test suites, a maximum of ten refinement rounds, and the same prompt-tuning budget, namely an equal number of prompt revisions on a held-out development split. We report results both at a fixed round cap and at a matched token budget. The matched-token view matters because memoryless configurations are cheaper, and a fair comparison must credit or penalize that.

The key ablation is the developer-only ledger at identical total memory tokens. It isolates the claim that placement of memory matters. A second ablation replaces the verified ledger with a raw transcript in the reviewer's context. It isolates whether gains come from asymmetry or from test-based filtering.

Beating the strongest method means reaching Adversarial Review's pass rate within one standard error on SWE-bench Verified and LiveCodeBench while using at most half its tokens. It also means showing a lower oscillation rate and a lower regression rate at equal rounds. If we win on tokens but lose on pass rate, we report that as a cost–accuracy trade-off, not a win.

Why not just…?

Why not just make every agent memoryless and keep only the latest code and error? This is the cheapest option and a real threat, because on short tasks stale history may be pure noise. Its known weakness is that a stateless loop cannot tell it is revisiting a failed state; STARK's reliance on a backtracking tree (STARK) is indirect evidence of this. The memoryless-for-both configuration is run directly. If it matches the reviewer-only ledger on oscillation and pass rate, the asymmetry hypothesis is refuted for these benchmarks.

Why not just add a non-model filter that rejects any program already seen? A hash-set blocklist of past programs catches exact revisits for free, and it may explain much of the benefit. It cannot catch semantic oscillation, meaning the same bug reintroduced in different code, a fix direction re-proposed after it already caused a regression, or alternation between two readings of the specification. We run the memoryless loop plus this filter as a baseline. We then split oscillation into exact and semantic revisits, where semantic revisits are detected by previously fixed tests failing again. The ledger must win on the semantic category to justify itself. If it does not, the honest conclusion is that memory only needs to be a hash set.

Why not just port debate memory masking symmetrically, with test outcomes as the filter? The debate result (Multi-Agent Debate with Memory Masking) suggests that filtering bad history, not asymmetry, may be the active ingredient. This is the closest alternative. The symmetric-ledger and symmetric-masked configurations test it head on. Our claim survives only if the reviewer-only ledger beats the symmetric ledger on regressions or over-editing, or matches it at clearly lower cost. The game-theory divergence result predicts the first outcome, but we treat that prediction as open.

Why not just use one long-context agent with Reflexion-style self-repair? Evidence in the file argues against single agents on these tasks. STARK reports kernels up to 16 times faster than a Reflexion agent on KernelBench Level 2, AgentCoder attributes its gains to separating test design from coding, and SWE-bench documents models being distracted as more context is added. We still include a single-agent self-repair loop with full history on LiveCodeBench as a compute-matched point, so the multi-agent overhead is justified by our own numbers.

Proposed Experiments

The main testbeds are LiveCodeBench, SWE-bench Verified, and SlopCodeBench, with HumanEval and MBPP-ET as ceiling sanity checks. LiveCodeBench provides problems less likely to be contaminated and short enough to run all six configurations with multiple seeds. SWE-bench Verified tests whether a memoryless developer survives the cross-file coordination needs that motivated MAGIS; there, developer "memorylessness" covers refinement history only, not repository context. SlopCodeBench measures erosion across successive changes to the specification. No reported result was found on Lacuna for any baseline on SlopCodeBench, so all numbers there will be our own.

We report five metrics:

  • final pass rate;
  • the pass-rate curve over rounds 1 to 10, to see whether the three-to-five-round plateau is delayed;
  • oscillation rate, split into exact and semantic revisits;
  • regression rate on previously passing tests and cumulative diff size, together measuring over-editing;
  • total tokens.

We use at least three seeds per configuration and bootstrap confidence intervals over tasks.

The decisive checks follow from the claim. The first is that the reviewer-only ledger beats the developer-only ledger at equal memory tokens on oscillation and regression rate. If the two tie, placement does not matter and the central claim fails. The second is that the reviewer-only ledger matches reimplemented Adversarial Review on pass rate with at most half the tokens. The third is that it beats the memoryless-plus-hash-filter baseline on semantic oscillation. The fourth, from the sweep over $k$, is that a ledger holding a single round captures most of the gain of an unlimited one, as the depth-one game-theory result would suggest. If longer memory is strictly better, that result's intuition does not transfer. The simulation study is read alongside the fourth check, to judge whether any failure to transfer comes from language-model behavior or from the game-theoretic intuition itself.