When a language model fixes code in a loop, it writes a program, runs a compiler or checker, reads the error, and tries again. Today the error message is pasted back into the prompt as text, and the model is free to ignore it. Models often write the same broken identifier or the same failing proof step again, round after round. This proposal treats the compiler's output as a constraint on generation. Each diagnostic is converted into a small, local restriction on which tokens the model may emit at the place the error points to. There are two kinds of restriction. A positive one limits an unknown identifier to the symbols actually in scope there. A negative one forbids the exact continuation that already failed at that location. Everywhere else the model decodes normally.
The underlying mechanism is dynamic vocabulary pruning: shrinking, at each generation step, the set of output tokens the model is allowed to choose from. So far that set has been built from translation dictionaries, recent tokens, or static grammars. Here it is built from execution feedback. The bet is that information the model reliably ignores when it arrives as prompt text becomes effective when it arrives as a mask. The payoff would be fewer rounds to a passing program, fewer repeated failed patches, and a later performance plateau on repository completion and Lean proof repair. The central experiment is designed so that a win cannot be explained by the extra information alone.
Core Claim
Holding the information fixed, delivering compiler feedback as a span-scoped decoding mask should beat delivering it as prompt text. We expect the gain to show up on two failure modes that iterative repair is known to suffer from. The first is hallucinated or out-of-scope symbols in repository code. The second is "thrashing," where the model cycles through equivalent invalid patches. The mask has three parts. It is scoped: it applies only inside the region the diagnostic localizes. It is two-sided: it both allows in-scope symbols and bans failed continuations. It is self-relaxing: it lifts itself when it forces the model into text the model considers very unlikely.
Where This Sits
Iterative repair systems motivate the problem. COCOGEN runs pylint, uses abstract-syntax-tree and CodeQL queries to find missing project definitions, and adds them to the prompt. It reports an 80% relative gain for GPT-3.5-Turbo on CoderEval and CrossCodeEval, with most errors fixed in one or two rounds. We reuse its retrieval step unchanged but route its output into the decoder instead of the prompt. Two other results establish the plateau we target. VERINA finds that o3 reaches only 4.9% pass@1 on Lean proof generation, and 64 rounds of compiler feedback raise this to about 20% at very large token cost. Yan et al. report sharp plateaus in iterative code translation. SWE-bench shows that adding more text context can hurt localization. That is our main reason to doubt that the prompt is the right channel for feedback.
Constrained decoding supplies the method and also limits the design. Type-Constrained Code Generation prunes tokens that cannot lead to a well-typed TypeScript program. It improves repair pass@1 by 34.1% on MultiPL-E/MBPP at 34.0% runtime overhead. Correctness-Guaranteed Code Generation builds a tree of parsers for a typed Lua dialect and contributes token healing across subword boundaries, which we need because identifiers split across tokens. Earley-Driven Dynamic Pruning makes grammar masking fast through state pruning and mask caching. All three build masks from a static language specification and have no memory of earlier attempts. Our masks come from the diagnostics of the previous round, so they can encode facts no grammar contains, such as "this lemma name does not exist in this environment" or "simp already failed on this goal."
Both constrained-decoding papers also report that hard masks can trap the model in degenerate loops. We take that as a constraint, and it is why the relaxation rule below exists.
On the vocabulary-pruning side, NanoSpec builds a context-derived sub-vocabulary of fewer than 3k tokens for speculative drafting, a speed-up technique in which a small model proposes tokens for a large one to check. It treats code as ordinary text and uses no feedback. Taming the Tail prunes low-probability tokens for stability in reinforcement learning. We borrow the view that cutting the tail can help accuracy, not only speed. We do not claim a wall-clock gain from a smaller output layer. NanoSpec itself notes that the output layer's share of runtime shrinks as models grow, and our masks are active only on short spans.
Research Program
After each failed round, a diagnostic compiler maps each error to a span and a mask. We handle three families.
The first family is name and attribute errors, such as undefined names, missing attributes, and unknown Lean identifiers. For these, the scope table (from the Python abstract syntax tree plus COCOGEN-style project queries, or the Lean environment) gives an allowed identifier set. That set is compiled into a trie over the model's tokenizations, with token healing handling subword splits.
The second family is failures at a specific location, such as a failing tactic at a given goal, a type mismatch at an argument, or a test failure pinned to a line. For these, the previous round's continuation at that location is added to a failure trie. Any path that exactly reproduces a banned continuation is masked. After normalization, the same applies to its whitespace- and alpha-renaming-equivalent variants.
The third family covers everything else. The span is left unconstrained, and the diagnostic still goes in the prompt.
The model regenerates the file or proof with the previous attempt and diagnostics in the prompt, exactly as in the text-only loop. The difference is that when decoding enters a flagged span, detected by aligning against the previous attempt, the positive and negative masks turn on.
To avoid trapping the model, we track the probability mass the unmasked model assigns to the allowed set,
$$m_t = \sum_{v \in A_t} p_\theta(v \mid x_{<t}),$$
where $A_t$ is the allowed token set at step $t$ and $p_\theta$ is the model's unconstrained next-token distribution. If $m_t < \tau$ for $k$ consecutive steps, the positive mask is dropped for the rest of that span and only the failure ban is kept. The threshold $\tau$ and window $k$ are tuned on a held-out split.
Low allowed mass is itself a signal. It suggests the diagnostic is pointing at the wrong fix, for example that the right move is to import a new module rather than pick an in-scope name. We log it so we can analyze when compiler feedback misleads.
The implementation is a logits processor on vLLM or Hugging Face for open models (Qwen2.5-7B, Llama-3-8B), plus the APRIL-style fine-tuned Lean repair models. Closed APIs cannot be masked, so all comparisons use open weights.
Baselines
The naive reference point is resampling. The model sees the same prompt as in round one, plus the failing code, and samples again under the same round budget. Any loop that fails to beat this is not using feedback at all.
The strongest competitor on Python repository completion is text-channel feedback: COCOGEN's full Generate-Verify-Retrieve-Fix loop, re-run on the same open model. It is the strongest because it already injects exactly the retrieved symbol definitions and diagnostics we use. The only difference from our method is the channel. We run it with identical retrieval output, identical prompts, and the same round budget and sampling temperature, and tune both methods' decoding hyperparameters with the same number of held-out trials. COCOGEN's reported number is an 80% relative pass-rate improvement for GPT-3.5-Turbo on CoderEval and CrossCodeEval. The file reports no absolute pass rate, and no reported result was found on Lacuna for COCOGEN with an open 7–8B model, so we establish that number ourselves. Beating it means a higher pass rate at equal rounds that is statistically reliable across seeds, or equal pass rate in fewer rounds.
On TypeScript repair, the strongest competitor is Type-Constrained Code Generation, which reports a 34.1% pass@1 improvement on MultiPL-E/MBPP repair. We run it inside the same multi-round loop, and we also run its combination with our failure ban. The honest bar here is complementarity: our masks should add pass rate on top of type constraints, because a type automaton cannot know which well-typed patch already failed.
On Lean, the references are VERINA's iterative ProofGen: o3 at 4.9% single-shot and about 20% after 64 rounds. That is a different, much larger model, so we re-run its loop on our open models. The second reference is the APRIL 4B/8B repair models from Learning to Repair Lean Proofs, reported at 27.4% and 31–35% single-shot repair accuracy. Beating these means higher repair accuracy at matched rounds, or matching the 64-round success curve with substantially fewer rounds and generated tokens.
Four ablations isolate each ingredient: positive masks only, failure bans only, masks applied to the whole output instead of the flagged span, and masks without the relaxation rule. The last one directly tests whether the entrapment problem reported by the constrained-decoding papers reappears here.
Why not just…?
Why not just put the diagnostics and in-scope symbols in the prompt? This is the real threat, and the experiment is built around it. The COCOGEN baseline receives exactly the information our masks encode, so any gap we measure comes from the channel. Evidence from SWE-bench and the round-by-round plateaus in VERINA suggests models often fail to act on text they have been given, but that is suggestive, not proof. If text-channel feedback matches our method on CoderEval and CrossCodeEval, we will report that the prompt is sufficient for name errors. We would then narrow the claim to failure bans on thrashing, which a prompt cannot enforce.
Why not just use grammar- or type-constrained decoding? Those methods mask what the language forbids, not what this program's history has ruled out. Type-Constrained Code Generation also requires a full prefix automaton for a statically typed language, which Python repositories and Lean tactic scripts do not offer in the same form. On TypeScript, where it applies, we stack the two methods. If our masks add nothing on top of type constraints, that result bounds the contribution to languages without such automata.
Why not just break thrashing with higher temperature, a repetition penalty, or more samples? These change diversity everywhere, including the large unchanged parts of the file, whereas failure bans act only at the failing location. We run a sweep of temperature and repetition penalty, plus best-of-n resampling at matched generated-token budget. If broad diversity achieves the same drop in repeated-patch rate at the same pass rate, the negative mask is unnecessary and we will say so.
Why not just fine-tune on repair data, as APRIL does? Fine-tuning teaches the model to read diagnostics better but still leaves it free to ignore them at decode time. We apply our masks to the APRIL-style fine-tuned models as well. The two approaches should compose, and the experiment shows whether they do.
Proposed Experiments
The first decisive check is the channel comparison on CoderEval and CrossCodeEval with Qwen2.5-7B and Llama-3-8B. We report pass@1 after rounds one through five, the rate of hallucinated symbols per round, and the number of rounds to a passing program. For thrashing, we define the repeated-patch rate as the fraction of rounds whose edit at the flagged span is equivalent, after normalization, to a previous round's edit.
The second check is VERINA ProofGen using the released sunblaze-ucb/verina tasks, plus APRIL held-out repair. We plot success against rounds up to 64 and against total generated tokens, and ask whether failure bans move the plateau. Results are broken down by APRIL mutation type, since line mutations (13.5% in the source paper) are where localized masks should matter least.
The third check is the TypeScript stacking experiment against type-constrained decoding on MultiPL-E and MBPP.
Across all settings, we log how often the relaxation rule fires and how often a relaxed span ended up passing. This measures how often compiler feedback points at the wrong fix, a quantity no prior loop has measured. Efficiency is reported only as generated tokens and rounds, not as output-layer wall-clock time.
References
- Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
- VERINA: Benchmarking Verifiable Code Generation
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- SWE-bench: Can Language Models Resolve Real-world Github Issues?
- Type-Constrained Code Generation with Language Models
- Correctness-Guaranteed Code Generation via Constrained Decoding
- Earley-Driven Dynamic Pruning for Efficient Structured Decoding
- NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies
- Taming the Tail: Stable LLM Reinforcement Learning via Dynamic Vocabulary Pruning
- Learning to Repair Lean Proofs from Compiler Feedback