Sub-Case Memory for Multi-Agent Code Generation

Programs written by large language models get better when the model is shown a solved example of a similar problem. Systems that keep a growing library of past solutions and reuse them, called case-based reasoning, have recently been rebuilt around language models. They hit a surprising wall. Showing the model more than one past solution makes it worse, because the examples interfere with each other and crowd the prompt. As a result, each new problem can borrow from at most one old problem, even when the useful pieces are spread across many. We propose to store and retrieve past solutions at the level of sub-tasks instead of whole problems. A planning agent splits the new problem into parts. Each part then goes to its own agent, which sees only the one stored sub-solution most relevant to its part. Solved and tested parts are written back into the library.

The bet is that interference comes from showing the model whole cases, not from drawing on several cases. If that is right, splitting the work across agents should let a single answer draw on many past problems while every individual model call stays short. The library should also become more useful as it grows. If it is wrong, the extra structure buys nothing over retrieving the single best whole solution. Our experiments are designed to tell these two outcomes apart.

Core Claim

The case-based reasoning cycle has four steps: retrieve a similar past case, reuse its solution, revise it to fit, and retain the validated result. DS-Agent runs this cycle with language models on data science tasks, using a planner, a programmer, and a debugger. That paper also supplies the observation that motivates this proposal: several retrieved cases in one prompt interfere, so the deployed system uses exactly one. CodeSim splits code generation into planning, coding, and debugging agents, and its planner recalls one exemplar problem. But that exemplar is a single fixed reference, and nothing solved is ever retained. CodeSim also reports that tracing dynamic-programming state by hand in text overflows the prompt. CodeChain shows that sub-modules, meaning individual helper functions, are a useful unit to show the model. It clusters sub-modules from several sampled solutions and feeds the most typical ones back as examples. However, this happens within a single query, and everything is discarded afterwards.

Our claim is the conjunction these papers leave untested. Consider a library of tested sub-modules that persists across problems, where each sub-agent sees exactly one retrieved sub-case for its own sub-task. Under a fixed token budget per model call, accuracy should keep rising as the number of distinct past problems contributing to an answer grows. Under whole-case retrieval, by DS-Agent's account, accuracy peaks at one case.

The idea of storing intermediate abstract solutions comes from Stratified CBR. That work stored abstract plans from hierarchical search as indices for route finding and showed that higher-level matching cuts search cost. We carry this to language-model agents. The abstract level becomes the planner's decomposition, and the ground level becomes code.

Research Program

What gets stored. We build a sub-case library from a stream of training problems. Each entry is a sub-module that passed tests. It records the planner's one-line description of the sub-task, the function signature, and the code. It also records provenance: which problem the sub-module came from, and whether the full solution passed.

How a new problem is solved. The planning agent first retrieves whole-problem cases. It uses these only for their decompositions, so it sees abstract plans, not code. It then writes a plan for the new problem, listing named sub-tasks with signatures.

Each sub-task goes to a coding agent. That agent receives the sub-task description, the signatures of neighbouring sub-tasks, and the single nearest sub-case, retrieved by embedding similarity over sub-task descriptions. We cap every coding call at a fixed token budget.

An assembly-and-debug agent then combines the parts and runs the visible tests. When a test fails, it sends the failure back to the specific sub-task responsible, in the spirit of CodeSim's debugging agent.

What gets retained. Retention writes sub-modules from solutions that pass the hidden tests back into the library. We also keep a lightweight form of the "boundary entries" from OBAM. OBAM stores explicit if-then rules for near-identical cases whose outcomes conflict, and it improves streaming diagnosis accuracy. In our setting, a boundary entry arises when two retrieved sub-modules with nearly identical descriptions differ in test outcome. We store a short note saying what distinguished the working one. This note is an ablation, not a core ingredient.

What is left out, and why. We deliberately do not pursue steering agents with stored latent "task vectors." Prior work shows that few-shot prompts compress into such vectors (In-Context Learning Creates Task Vectors). OBAM's results, however, suggest that the fine distinctions between near-identical cases are exactly what compression risks erasing. Code also needs verbatim detail.

What we analyze. Three analyses carry the claim. The first is accuracy as a function of how many distinct source problems an answer drew on. The second is the learning curve as the library grows over the stream. The third is the largest single-call context, compared against monolithic prompting at matched total tokens.

Baselines

Naive reference. The do-nothing reference is the same backbone model given the problem directly, with no exemplar and no agents. A second naive point is the same model with chain-of-thought followed by a self-repair loop on visible tests. Both use the same sampling budget.

Strongest existing method. The strongest competitor is CodeSim, rerun with its released prompts and its adaptive planning and debugging loop on our backbone. For reference, CodeSim reports pass@1 of 29.1% on CodeContests and 22.0% on APPS (CodeSim). The file pairs GPT-4o only with its HumanEval figure of 95.1%, so we treat the CodeContests and APPS numbers as reference points and rerun everything ourselves.

CodeChain is the strongest sub-module method. It reports GPT-3.5 pass@1 on APPS rising from 22.33% to 30.24%, and a 76% relative gain on CodeContests (CodeChain). We run it with the same number of samples per problem.

The decisive competitor. The most important baseline is one we construct, because it isolates our key ingredient. It is CodeSim with whole-problem retention. Every solution that passes is retained as a complete case, and the planner retrieves the single best whole case. This is essentially the DS-Agent recipe transplanted to competitive programming. It receives the same training stream, the same library-building budget, and the same retriever. This baseline differs from our method only in the granularity of what is stored and in how retrieved material is split across agents.

Fairness and what counts as winning. All methods share one backbone, one total token budget per problem (measured, not assumed), the same visible tests, and an equal prompt-tuning budget of a fixed number of development-set trials each. Beating the field means a pass@1 gain over whole-case retention that holds across three seeds on both APPS and CodeContests, at equal or lower total tokens.

Why not just…?

Why not just retain whole solved problems and retrieve the best one? This is the real threat, and it may work as well as our method on problems that have close whole-problem neighbours. DS-Agent's single-case result says it will not benefit from a second case. It says nothing about whether a second case's parts would help. The whole-case retention baseline above settles this directly. We additionally split the test set by how close each problem is to its nearest whole training problem. Our claim predicts the gap opens on problems with low whole-problem similarity but high overlap with individual sub-tasks. If no such gap appears, the claim is false.

Why not just put many retrieved cases into one long prompt? DS-Agent reports interference from multiple cases. Long-context studies such as KALM and MedAlign report that models lose track of distant details. We test this rather than assume it. We add a single-agent baseline that receives the same set of sub-cases our agents see, concatenated into one prompt of equal total size. This is the cleanest test of whether splitting the work across agents, rather than the retrieved material itself, matters.

Why not just use flat retrieval over code snippets with one model? Chunked retrieval ignores the planner's decomposition. OBAM's results show that flat retrieval falls apart when near-identical items have different outcomes. We run a baseline that retrieves the top sub-modules by similarity to the whole problem statement and gives them to a single coder. Comparing it with our method shows whether keying retrieval on the plan matters.

Proposed Experiments

We use the APPS and CodeContests training splits as the stream from which libraries are built. We evaluate on their test splits, processed in a fixed random order, with retention switched on. HumanEval and MBPP serve only as sanity checks, since CodeSim already reaches 95.1% and 90.7% on them. Contamination of the backbone's pretraining with test solutions is a risk. We address it by also reporting results on CodeContests problems dated after the backbone's training cutoff, where available. The metrics are pass@1, total tokens per problem, largest single-call context, and the number of distinct source problems contributing to each answer.

Four checks decide the outcome. The first is the source-count sweep. We vary the maximum number of distinct source problems from one upward, holding per-call budget fixed. We expect our method's accuracy to keep rising while whole-case retrieval stays flat or falls. The second is a shuffled control. Retrieved sub-cases are replaced with random tested sub-modules of matched length, which rules out the explanation that any extra code helps. The third is a retention ablation, comparing a frozen library against an online one. We expect a learning curve that rises with stream position only when retention is on. The fourth focuses on dynamic-programming problems, which we pick out by their problem tags. CodeSim's own analysis places its weakness there. If bounded per-agent contexts help anywhere, they should help most on that subset.

A negative result on the first check, together with parity against whole-case retention, would show that interference is about the number of cases rather than their granularity. That would itself be a useful correction to how case libraries for language models are designed.