Compiler-Certified Generative Replay for Continual Code Models

When a code model is fine-tuned on one new task after another (a new language, a new library, a new kind of edit), it forgets the earlier ones. The usual remedy is to keep a buffer of old training examples and mix them back in. For code, that buffer is often exactly what cannot be kept, because it is proprietary, under a license, or under an agreement that forbids retaining it. Generative replay offers a way around this. The model writes its own imitation examples of past tasks and rehearses on those. In vision, however, generative replay is known to decay like a photocopy of a photocopy. Each round's generator learns from the previous round's flawed samples, the flaws compound, and long task sequences end in noise. This proposal starts from one observation. Code, unlike images, comes with external judges that do not drift: compilers, type checkers, and the ability to run two programs on the same input and compare their outputs. We propose to place these judges, together with an iterative repair loop of the kind used in LLM self-debugging, between the model and its own replay data. No synthetic program is rehearsed unless it has been certified, and failing samples are repaired rather than discarded.

The central claim has two parts and can be tested. First, certification stops compounding error along the properties the judge checks, but not along the properties it ignores. This yields a sharp prediction: across a long task chain, compile rate and translation equivalence of replayed data should stay flat, while unchecked properties such as fidelity to natural-language intent keep decaying. Second, repairing failed samples matters beyond simply filtering them out. Filtering alone keeps only the easy programs, which shrinks the replay set toward short, trivial code. That shrinkage is itself a form of the photocopy collapse. Iterative repair keeps the hard examples in the replay set. If the second claim fails at equal compute, the method reduces to a much simpler one, and we will report that.

What Is Known and What Is Missing

The motivating evidence comes from continual learning for image generation. Generative Models from the perspective of Continual Learning shows that when a model treats its own outputs as ground truth, it enters a "snowball" feedback loop that collapses on harder data such as CIFAR-10. Memory Replay GANs and Lifelong GAN document how early generation errors propagate forward and strip coherence from replayed distributions. Learning to Remember and Lifelong Generative Modelling Using Dynamic Expansion Graph Model diagnose the root cause: nothing anchors the generator to the true distribution, so it optimizes for its own idiosyncrasies. The strongest fix in that line, DDGR, steers diffusion sampling with a classifier's gradient. That classifier is itself a network being updated, so the anchor can drift too. DDGR also costs roughly 75,000 seconds of training on CIFAR-100, against about 8,000 for regularization methods. Our proposal replaces a learned, drifting anchor with a deterministic, non-differentiable one, which is only possible because the data is executable.

The method comes from iterative code refinement, where a model generates code, runs it, reads the errors, and edits. DeepFix used compiler messages to repair C programs. CodeTransOcean reports that three rounds of compiler-feedback self-debugging lift GPT-3.5's execution success on translation from 48.57% to 52.57%. MEIC shows that rolling back to the last good version blocks the "new error trap," where fixing one bug introduces another. Two results constrain the design. VERINA (VERINA) and CodeTransOcean both show refinement plateaus after a few rounds, so we cap the loop at a small budget instead of relying on deep repair. Rethinking Code Refinement shows machine edits make code slower about 30% of the time, and Do Users Write More Insecure Code with AI Assistants? shows that feeding model outputs back into prompts can compound flaws. Repair can damage properties it does not check, which is exactly what our checked-versus-unchecked analysis is built to detect.

Three papers sit closest to this intersection. Exploring Continual Learning for Code Generation Models introduces the CODETASK-CL benchmark and evaluates two families of methods. The first is a real-data buffer (experience replay). The second is prompt pooling, which learns a pool of small soft prompts and selects among them by matching a learned key to each input; it fails through "key drift" unless oracle task identities route the prompts. That paper never tests generative replay, and it treats code refinement as a one-shot mapping rather than an execution loop. SERS does self-generated rehearsal for instruction-tuned LLMs. It fights drift by having the frozen base model fill in templates built from a 1% real-data seed and by blending the base and tuned models' predictions. SERS checks nothing by execution, needs that seed, and is evaluated only on natural-language tasks with ROUGE-L. MePo builds pseudo task sequences from pretraining data for vision classifiers and involves no executable artifacts. What none of them does is gate self-generated replay with an external execution oracle and an iterative repair loop. None measures whether this stops compounding error across task generations.

Core Mechanism

The learner is a code model trained on a sequence of tasks with no stored real data. Before training on task $t+1$, the current model generates pseudo-examples for each earlier task, prompted only by the task's instruction format. Each pseudo-example then passes through a judge specific to its task.

For generation and bug-fix tasks in Java, the judge is the compiler and type checker. For a bug-fix pair, it also requires that the "fixed" side compile and differ from the "buggy" side. For translation pairs, the judge requires that both sides compile. It then runs both programs on inputs the model generates for itself and checks that their outputs agree. This differential test needs no stored test suite.

A sample that fails is returned to a repair step along with the judge's message. The repair step allows at most $K = 3$ rounds, since gains plateau beyond that in CodeTransOcean. Following MEIC, it rolls back if an edit introduces new errors. Samples still failing after $K$ rounds are dropped. The certified set is mixed with real task $t+1$ data for training.

The repairer is either the current model or the frozen initial checkpoint. It is never a stronger external model, because that would smuggle in knowledge the learner never had.

Summarization (Ruby to English) has no execution judge. This is deliberate: it serves as the within-experiment control for unchecked drift.

Research Program

The work has three parts. The first builds the certification and repair pipeline for Java, C#, and the execution-based subset of CodeTransOcean.

The second runs the method on two task streams. One is the four-task CODETASK-CL sequence, for direct comparison with published numbers. The other is a longer chain of eight to ten language-pair translation tasks drawn from CodeTransOcean. Four tasks are too few to expose photocopy decay, which needs many generations to appear. The long chain is evaluated by execution success on the LLMTrans subset.

The third is a diagnostic study that tracks the replay set itself across generations. For each past task, at each stage, we record the following:

  • the raw pass rate before repair;
  • the pass rate after repair;
  • structural diversity, measured as distinct AST n-grams;
  • difficulty, measured by program length, nesting depth, and the number of distinct API calls;
  • distance to the held-out real distribution, measured as a distance between embeddings of replay samples and real samples.

The prediction is specific. Without certification, raw pass rate should fall with each generation. With filtering only, pass rate holds but diversity and difficulty fall. With repair, all of them should stay closer to the real distribution.

Learners are CodeT5 (220M), for comparability with CODETASK-CL, and DeepSeek-Coder-1.3B. The larger model can actually use compiler feedback, and the same family is used as a code judge in Rethinking Code Refinement. CodeT5 may be too weak to repair its own samples. For CodeT5 we therefore also report a variant in which the frozen initial DeepSeek checkpoint repairs, and we label it clearly as having a different capability profile.

Baselines

The naive reference point is plain sequential fine-tuning with no protection against forgetting. It sets the floor for forgetting on both streams and is trained with the same optimizer, learning rate schedule, and steps per task as every other method.

The strongest existing method on CODETASK-CL is experience replay with a real buffer. Exploring Continual Learning for Code Generation Models reports 50.26 average BLEU for CodeT5 with experience replay. It is the strongest competitor because it rehearses true data, which is the upper bound any buffer-free method is chasing. We reproduce it with buffer sizes matched to our number of replay samples. Beating it is not our success criterion, since it uses data our setting forbids. We count the method as successful if it closes most of the gap between sequential fine-tuning and experience replay.

The strongest buffer-free method in that paper is prompt pooling with teacher forcing, reported at 44.33 average BLEU with forgetting 1.68. It needs oracle task identities during training. We run it with the published configuration and count our method as beating it if we exceed 44.33 average BLEU without task identities. We also report forgetting alongside.

SERS-style self-rehearsal without execution is the most direct competitor in spirit. We re-implement its base-model template filling and prediction blending on code, with its 1% seed. No reported result found on Lacuna for SERS on any code benchmark, so we report our reimplementation's numbers next to its published SuperNI setting for transparency. It isolates whether execution certification adds anything beyond statistical anchoring to the base model.

Two ablations isolate the key ingredient. The first is unchecked self-replay, in which the same generator is used and every sample is rehearsed. The second, and more important, is filter-only replay at matched compute. Here failing samples are discarded and replaced by fresh samples, with the generation budget equal to the total model calls the repair loop spends, so the two methods produce certified sets from the same number of model calls. Repair counts as mattering only if it beats filter-only replay at this matched budget on final task performance or on the diversity and difficulty diagnostics.

Why not just…?

Why not just keep a small real buffer? Experience replay is the best performer on CODETASK-CL at 50.26 BLEU. Where data retention is allowed, it is the right tool, and we do not claim otherwise. This proposal targets the setting where old code cannot be retained, which the research file attributes to licensing and confidentiality constraints. It also asks a separate scientific question that a buffer cannot answer: whether an external oracle can stop compounding error. We report the remaining gap to experience replay honestly at several buffer sizes, including the smallest buffer that matches our method.

Why not just isolate task-specific parameters with prompt pooling or adapters? Exploring Continual Learning for Code Generation Models shows that unrouted prompt pooling collapses to 22.79 BLEU with forgetting 27.43 because of key drift. The fix needs oracle task identities and still trails experience replay by about six BLEU. Our method needs no task identity at training or test time. We run prompt pooling with teacher forcing as a baseline. If it matches us on CODETASK-CL, the long translation chain, where task boundaries are less clean, is where the comparison will be decided.

Why not just filter by compilation and throw failures away? This is the most serious threat, because it is cheaper and might be enough. The case for repair rests on the argument that filtering biases replay toward programs the model already finds easy, which reproduces photocopy collapse in a new form. That argument is a hypothesis, not a result. The matched-compute filter-only ablation and the diversity and difficulty diagnostics settle it directly. If filtering matches repair, the publishable finding becomes "execution filtering suffices," and we will state it that way.

Why not just prompt the frozen base model for replay, as SERS does? A frozen generator does not compound its own errors, but it also never knows the new tasks the learner has acquired. Its raw code is often wrong: VERINA and MAGIS report low one-shot success for strong models on hard code tasks. We test this by running our pipeline with the frozen base model as generator. If certified base-model replay works as well as certified self-replay, then the compounding effect is not the bottleneck in code, and that result is informative either way.

Decisive Checks

The first check is the checked-versus-unchecked split on the long translation chain and on CODETASK-CL. Under certified replay, forgetting on execution-judged properties should be roughly flat across generations. Forgetting on summarization and on BLEU against reference implementations should continue to grow at close to the unchecked-replay rate. If certified properties also decay, the oracle is not breaking the feedback loop and the core claim fails. If unchecked properties stop decaying too, certification has a broader regularizing effect worth explaining.

The second check is filter versus repair at matched compute. The method needs repaired replay to keep difficulty and structural diversity closer to held-out real data and to yield lower final forgetting.

The third check is a regression audit. We use the efficiency judge from Rethinking Code Refinement and a check of whether the edit removed the bug. These test whether repaired replay samples "pass" by degenerate means, such as deleting logic or making trivial edits. Those are the unchecked side effects that refinement is known to produce.

Headline numbers are average BLEU and forgetting on CODETASK-CL, for direct comparison with the published 50.26, 44.33, and 22.79, and execution success on CodeTransOcean's LLMTrans subset after each task in the long chain. Each is reported over three seeds.