Feedback-Contrastive Verification for Code Repair Loops

When a language model fixes code over several rounds, it writes a program, runs it, reads the error or the rendered output, and tries again. A common and costly failure is that the model reads the feedback and then writes the same broken line again. We propose to catch and correct this "stubborn repair" almost for free, using work that a fast decoding method already does.

Speculative decoding speeds up generation by letting a cheap source propose tokens that the large model checks in one pass. For code edits, that cheap source can simply be the previous version of the program. When the large model checks the old program after seeing the new feedback, it produces the probability of every old token given the feedback. The previous round already produced the probability of those same tokens without that feedback. Comparing the two tells us, token by token, where the feedback should have changed the model's mind but did not. We then use that comparison to push the model off lines it is repeating out of habit.

The main testbed treats feedback as a perceptual signal rather than text. The model writes an audio-effects chain, the chain is rendered to sound, and the rendered audio is projected back into the model's input space as continuous embeddings. This is the regime where a model's prior beliefs are most likely to override what it is shown.

Core Claim

Each round of an iterative repair loop contains two conditional distributions over the old code: one without the newest feedback, cached from the round that produced the code, and one with it, computed during speculative verification. Contrasting them gives an approximate "context versus memory" signal of the kind used against knowledge conflicts, at roughly the cost of one pass instead of two.

Using this signal inside the verification step should lower how often the model repeats a region that the feedback implicated. It should also raise repair success per round, while keeping most of the speedup of reuse-based drafting. The claim is falsifiable in two directions. If exact two-pass contrastive decoding gives much better repairs than our cached approximation, the approximation is the weak link. If simply dropping the old code from the prompt removes stubbornness just as well, the mechanism is unnecessary.

Where This Sits in Prior Work

Three lines of work supply the pieces. None combines them.

Reuse-based drafting for code edits. EFFICIENTEDIT shows that code edits reuse most of the original snippet. It drafts with the unedited code, lets the target model's rejections locate the edit, and switches to a trained edit drafter at those points. It reports a 10.38x speedup on CanItEdit and 13.09x on CodeIF-Bench with DeepSeek-Coder-33B. It works in a single turn, without execution feedback. Its acceptance rule accepts old tokens whenever the target is comfortable with them. In a repair loop that rule is a liability, because it will happily re-accept a buggy line the target is stubborn about. We reuse its drafting and change what counts as acceptance.

Decoding-time fixes for context–memory conflict. Adaptive Contrastive Decoding contrasts a model's token distributions with and without retrieved context. It weights the contrast by the two entropies. On the noisy TriviaQA subset with Llama 2 7B it reports 42.06 exact match against 24.75 for standard context-aware decoding. Its stated cost is two forward passes per token.

Speculative Contrastive Decoding shows that speculation and contrast fit together. Easy tokens are accepted quickly and uncertain ones get contrastive correction. It raises Llama-2-70B HumanEval Pass@1 from 28.66% to 37.20% with a 1.78x–3.32x speedup. However, its contrast is between a large and a small model, not between the model with and without external evidence.

We take ACD's with/without-context contrast and use the speculative structure noted by SCD to make it nearly free. In a repair loop, the "without" pass already happened in the previous round.

Evidence that repair loops need this. Three findings motivate the proposal. Gains from compiler and traceback feedback saturate after a few rounds (Yan et al.). Iterative prompting can compound initial errors (Srivastava et al.). MMKC-Bench finds that multimodal models prefer their internal beliefs over conflicting visual evidence, and that this preference grows from 3B to 72B parameters. The same benchmark reports conflict-detection accuracy around 75–79%. That result constrains our design: models can often tell that evidence conflicts with their beliefs and still not act on it. This is why we intervene in decoding rather than only in the prompt.

OpenCodeInterpreter supplies the text-only repair setting. On HumanEval its 33B model goes from 79.3 Pass@1 single-turn to 82.9 with execution feedback.

The perceptual application. LLM2Fx-Tools projects audio features, from an audio encoder through a small transformer adapter, into a language model's input space. The model then writes executable effect chains for the Pedalboard audio library. It reports 80% planning accuracy and a Spearman correlation of 0.56 for effect ordering. It runs one shot: the rendered audio is never fed back. We close that loop.

Research Program

The loop. The loop starts from a dry track and a reference wet track, meaning the target sound. The model writes an effect chain, and Pedalboard renders it. The rendered audio is encoded by the same audio encoder and projection adapter as the inputs. These continuous embeddings, which the model reads as if they were soft tokens, enter the next round's prompt with any runtime errors. The model then revises the chain.

A text-only twin of the loop, run on HumanEval and CanItEdit with interpreter tracebacks, tests whether the effect depends on the perceptual setting.

Verification rule. In round $r$, the target model generated the old program $y$. We cache $\ell^{0}_j$, the target's top-$k$ log-probabilities at each old position $j$ (a few kilobytes per program). In round $r+1$ the old program is proposed as the draft and verified under the new context, which includes the feedback, giving $\ell^{f}_j$. Verification uses the adjusted scores

$$\tilde{\ell}_j = \ell^{f}_j + \beta_j\,(\ell^{f}_j - \ell^{0}_j), \qquad \beta_j = \lambda\,\frac{H^{0}_j}{H^{0}_j + H^{f}_j},$$

where $H^{0}_j$ and $H^{f}_j$ are the entropies of the two distributions at position $j$, and $\lambda$ is a single tuned scale. The weight follows ACD's rule: feedback gets more say where the model was unsure without it. An SCD-style plausibility filter restricts $\tilde{\ell}_j$ to tokens that are reasonably likely under $\ell^{f}_j$, so the contrast cannot promote nonsense.

Effect on acceptance. An old token is accepted only if it survives under $\tilde{\ell}$. Tokens the feedback did not affect have $\ell^{f}\approx\ell^{0}$ and pass through at full speed. Tokens whose probability the feedback lowered, even if not enough to flip the plain argmax, are rejected. At those tokens generation hands over to the drafter.

Drafter. The drafter is a small model that receives a compressed, pooled summary of the audio embeddings, in the spirit of the single-token visual adaptor in ViSpec. A text-only drafter cannot follow a perceptually grounded target (MASSV).

Known approximation. The cached $\ell^{0}_j$ was computed with the old prefix and without the old program in context. The new pass has the old program in its history, which encourages copying. After the first edit, the prefixes also no longer match. The cached distribution is therefore a stale stand-in for "the model without this feedback." We measure the cost of that staleness directly against exact two-pass ACD rather than assume it is small.

Analysis. We will test whether the tokens flagged by $\ell^{f}-\ell^{0}$ coincide with regions the feedback actually implicates: the traceback line in code, or the parameter responsible for the largest spectral mismatch in audio. As a diagnostic, we will also check whether stubborn tokens show the memory-head attention signature described by Jin et al..

Baselines

Do nothing. The naive reference is plain autoregressive multi-turn repair with the same target model, prompt, and number of rounds, following the OpenCodeInterpreter recipe. It sets both the latency and the repair-rate floor. For the strongest text reference, OpenCodeInterpreter-33B reports 82.9 HumanEval Pass@1 with execution feedback (OpenCodeInterpreter). We will reproduce with the same model family, so our numbers are compared on our own runs rather than against the quoted figure.

Speed baseline. The strongest speed competitor is EFFICIENTEDIT applied every round: reuse drafting with the same edit drafter and its entropy-aware top-$k$ acceptance. It reports 10.38x on CanItEdit with DeepSeek-Coder-33B (EFFICIENTEDIT). It shares our drafter and our drafts, so the only difference is the acceptance rule. We beat it if repair success after a fixed number of rounds rises and the repeat rate falls, while wall-clock speedup stays within about 15% of its own on identical hardware.

Correctness baseline and key ablation. The strongest correctness competitor on the conflict question is exact ACD. It runs a second full pass per token with the feedback removed from the prompt but the old program still present, which is the "true" without-feedback distribution. On its original task it reports 42.06 EM against 24.75 for context-aware decoding (ACD). No reported result for ACD in code repair or audio refinement was found on Lacuna.

This comparison isolates our key ingredient, the cached distribution. We claim success if we recover most of exact ACD's repair gain at a fraction of its cost. A second ablation keeps our pipeline but sets $\lambda=0$, which reduces to EFFICIENTEDIT-style verification. A third removes the entropy weighting (fixed $\beta$).

Perceptual baseline. On the audio task, the reference is single-shot LLM2Fx-Tools, with reported 80% planning accuracy and 0.56 ordering correlation (LLM2Fx-Tools), plus our plain iterative loop without contrast. No reported result for iterative refinement on the LP-Fx data was found on Lacuna.

Fairness conditions. All methods get the same target model, drafter, number of rounds, sampling temperature, and a matched tuning budget. The budget is a small grid over $\lambda$, and an equal-size grid over the top-$k$ and entropy thresholds of competing methods, tuned on a shared held-out split.

Why not just…?

Why not just rerun EFFICIENTEDIT each round? It gives nearly all of the speed, and that is exactly the problem. Its acceptance rule is built to keep old code whenever the target tolerates it. In a repair loop, that preferentially re-accepts lines the model is stubborn about. The $\lambda=0$ ablation is this system inside our pipeline. If it matches our repair rate, the contrast adds nothing and we will say so.

Why not just use exact contrastive decoding (ACD) with and without the feedback? It is the principled version of our signal and may repair better. ACD's own paper names the cost: two forward passes per token (ACD), and those passes cannot share reuse drafting as cleanly. This is a real threat. The head-to-head comparison settles it by measuring how much of ACD's gain survives the cached approximation, per unit of wall-clock time. If the gap is large, we will report that a cheap refresh of $\ell^{0}$ at edited regions is needed.

Why not just drop the old code from the prompt, or tell the model the line is wrong? Removing the old program removes the copying pressure that plausibly drives stubbornness. Pointing at the traceback line is a Chain-of-Note-style prompting fix (Chain-of-Note).

The evidence against prompting alone is indirect. MMKC-Bench finds that models detect conflicts yet still follow their priors (MMKC-Bench). ContextFocus reports that activation steering beats prompting and stacks with it (ContextFocus). Restarting from scratch also discards the reusable 70% that makes speculation fast, and it may lose correct partial fixes.

We will run both as baselines, and also combine each with our rule. A from-scratch restart that matches our repair rate at comparable latency would falsify the need for the mechanism.

Why not just describe the rendered audio in text instead of projecting it? Cascades are strong: a transcribe-then-reason pipeline ranked 6th of 17 systems in AHELM. For effect chains, one can compute loudness, spectral-centroid, and decay-time differences between the rendered and reference audio and state them as numbers. This is a serious competitor. It also turns our method into a pure text-loop method, which is fine, because the verification rule does not care how the feedback is encoded.

We will run this text-descriptor loop both with and without contrastive verification. The outcome tells us two separate things. The descriptor-versus-embedding comparison says whether the projected embeddings carry information that numbers miss, which LLM2Fx-Tools argues they do. The with/without comparison says whether the contrast helps in either encoding.

Proposed Experiments and Decisive Checks

Datasets. The text arm uses HumanEval with interpreter feedback over up to four rounds, and CanItEdit (CanItEdit) with its hidden unit tests turned into execution feedback. We add a small constructed API-drift set: HumanEval-style tasks run in an environment where a commonly used library function has been renamed or changed in behavior. This forces a direct conflict between what the model memorized and what the traceback says. No reported result exists for it on Lacuna.

The audio arm uses LP-Fx examples paired with MedleyDB audio. It measures planning accuracy, ordering correlation, and per-round spectral distance between rendered and reference audio.

Metrics. Four metrics are reported throughout: repair success after each round (Pass@1, or audio distance), wall-clock speedup, mean accepted draft length, and a stubbornness rate. The stubbornness rate is the fraction of rounds in which the feedback-implicated region is regenerated token-identical.

Decisive checks. Four results decide the outcome.

First, on the API-drift set, the stubbornness rate should fall clearly below both EFFICIENTEDIT and plain autoregressive repair. If it does not, the mechanism has failed on the conflict it targets.

Second, exact ACD should beat our cached version by only a small repair-rate margin while costing substantially more time. A large margin means the approximation fails.

Third, the gains should persist in the audio arm and grow with target scale, as MMKC-Bench's scaling trend predicts (MMKC-Bench). We test this by repeating the audio loop with a small and a large target from the same model family.

Fourth, the round-over-round curve should keep improving after the round-two or round-three plateau reported by Yan et al.. Pushing past that plateau is the practical payoff this proposal is betting on.