When a language model repairs code over several rounds, something must decide after each round whether to keep the new version, roll back, or stop. In current systems that decision often comes from a second model acting as reviewer, from a preference for small edits, or from a few visible tests. We propose that these acceptance rules repeat a failure already well documented for text-overlap metrics. They reward revisions that look like the previous version and penalize revisions that change the code a lot, whether or not the change fixes the program. If so, part of the familiar plateau after two or three repair rounds would come from the acceptance rule rather than the generator. The loop would throw away correct restructurings and keep one-token changes that break the program.
The concrete plan has three parts. We build stress sets in which surface similarity and correctness are deliberately pulled apart. We measure how strongly each acceptance rule tracks surface similarity versus held-out execution results. We then test a replacement rule, differential execution, which runs the old and new versions on generated inputs and accepts a change only when its behavior moves in the right place. The decisive outcome is whether swapping only the acceptance rule moves the plateau.
Core Claim
In iterative code refinement, the module that judges each revision behaves like a "lazy metric": its verdicts are explained better by how much the code changed than by whether it now works. This bias predicts specific failures. Repairs that need a large structural change get rejected or rolled back. Small breaking edits get accepted. Success curves therefore flatten early.
An acceptance rule that measures behavior change directly should keep improving past round three, and the gain should be concentrated on problems whose correct fix is far from the starting code in edit distance. Earlier work has documented each half of this separately but has not put them together.
Why This Connection, and What Prior Work Establishes
The evidence that surface similarity misleads comes from the n-gram literature, meaning metrics such as BLEU and ROUGE that count shared word sequences between an output and a reference.
Out of the BLEU compared BLEU, ROUGE-L, ChrF and CodeBLEU against developer ratings on CoNaLa and Hearthstone. It found that BLEU differences under 5 points (CoNaLa) or 2 points (Hearthstone) cannot be separated from noise, and that code-aware metrics did not reliably beat character overlap. CodeTransOcean reports that BLEU and CodeBLEU barely correlate with whether translated programs run. Both papers motivate this proposal, but both evaluate finished outputs against a reference. Neither looks at the round-by-round decisions inside a repair loop. Repair edits are typically a few tokens long, which is exactly the range where those papers show surface metrics carry no signal.
The sharper motivation is Spurious Correlations in Reference-Free Evaluation of Text Generation. It showed that learned faithfulness scorers for summaries largely track copying. DAE correlates 0.76 with "density", the average length of copied spans, but only 0.38 with human judgments. On faithful summaries written with little copying, FactCC and DAE fall to about 38% pairwise accuracy, below a naive density heuristic at 40.45%. Our hypothesis is that reviewer agents in code loops carry the same shortcut, with "similarity to the previous version" standing in for "copying from the source". We borrow that paper's method directly: build a subset where the shortcut and the truth disagree, and measure accuracy on it.
The repair-loop side supplies the setting and the failure symptoms. MEIC pairs a debugging agent with a scorer agent and rolls back revisions that regress. It reports 93% syntax and 78% functional fix rates on Verilog with GPT-4, and it names the "new error trap", where repairs introduce fresh bugs. CodeTransOcean's self-debugging loop lifts ChatGPT from 48.57% to 52.57% debugging success over three rounds and observes the plateau. MAGIS uses a QA-engineer agent as an in-loop reviewer. These systems supply the method we probe. None of them asks what their scorer or reviewer is actually responding to.
On the text side, SummExecEdit builds factual-error cases with deterministic substring edits. It finds that Claude 3 Opus detects errors at 0.67 but explains them correctly only at 0.49, and that it blames the wrong span 45.4% of the time. We borrow its construction idea, controlled small edits, to make code that is close to the original but broken. The span-misattribution finding constrains our design: we should not expect a model judge to localize which change caused a regression.
Some nearby work points in a different direction. SPADE and LMUNIT carry the idea of tests into checking text. We move the other way, carrying the text-metric critique into code loops, so these papers are context rather than competitors.
Research Program
The first component is a stress set of revision pairs. Each pair is a predecessor program and a candidate revision, labeled correct or incorrect by held-out execution. The pairs come from the Python portion of CodeTransOcean's LLMTrans setting and from the Verilog debugging tasks used by MEIC.
Pairs fall into four cells, defined by two properties: whether the revision is lexically close to or far from its predecessor, and whether it is correct or broken.
- Close and broken revisions are made with SummExecEdit-style single-token mutations, such as a flipped comparison, an off-by-one index, or a dropped declaration, applied to correct programs.
- Far and correct revisions are refactorings and alternative implementations, either generated by a model or taken from distinct correct solutions, and admitted only if they pass all held-out checks.
- Close-correct and far-broken revisions are harvested from real loop trajectories.
The key subset is the "disagreement" set, where the far revision is correct and the close one is broken. It plays the role of the abstractive-faithful subset in the Spurious Correlations paper.
Every check is split into a part the loop can see and a part held out for evaluation. Where a task ships only a single execution check, we add held-out inputs whose expected outputs come from the reference solution.
The second component is the diagnosis. For each acceptance rule, we fit how well its verdicts are predicted by surface similarity to the predecessor (ChrF and token edit distance) versus by held-out correctness. We also measure pairwise accuracy on the disagreement set against a pure "prefer the smaller edit" heuristic.
The third component is differential execution as the acceptance rule. We generate inputs, using the model plus random and boundary values, and run predecessor and candidate on them. The candidate is accepted only if two conditions hold. Its outputs change on inputs where the visible failing check implicates behavior. They stay unchanged elsewhere, unless the visible tests require otherwise.
This rule judges a revision by what it does, not how it reads. It also avoids needing correct expected outputs for the generated inputs, because it compares two versions rather than checking either against an answer. It is cheap enough to run every round. Rollback follows MEIC's rule, applied to this signal.
Baselines
The naive reference point is no refinement: the generator's first output, reported as success after zero repair rounds. For ChatGPT on LLMTrans, CodeTransOcean reports 48.57%. We rerun this with our own generator, since cross-model numbers are not comparable.
The first loop baseline is CodeTransOcean's raw compiler-feedback loop, which accepts every revision. Its reported ChatGPT figure is 52.57% after three rounds.
The strongest existing method is MEIC's design: a scorer agent plus rollback. MEIC reports a 78% functional fix rate with GPT-4 on its Verilog benchmark. It is the strongest competitor because it already has an explicit acceptance-and-rollback step, so any gain we show must come from what that step measures. We reimplement it with the same generator model, scorer prompts adapted from the paper, and the same round budget.
We also run three further acceptance rules. The first accepts a revision if the visible tests pass. The second prefers the smallest edit, which makes the suspected shortcut explicit. The third is a frontier model used as judge with the same token budget as our input generation.
All conditions share the generator, tasks, round limit (up to eight rounds, to expose plateaus), and total compute. Compute is counted as model tokens plus sandbox executions.
The key ablation keeps everything in our system but replaces differential execution with the MEIC-style scorer. This isolates whether the acceptance signal itself is the cause. A second ablation drops the localization condition, accepting any behavior change. This tests whether "changed in the right place" matters beyond "changed".
Beating the strongest baseline means two things. Held-out success must be higher at round eight, with a confidence interval excluding zero over task-level resampling. The success curve must also still be rising after round three, where MEIC-style acceptance has flattened. A gain only at rounds one to two would not support the claim.
Why not just…?
Why not just accept whatever passes the visible tests? This is the most direct objection, and it is serious when visible tests are thorough. Our claim rests on visible tests being sparse, which is common in the translation and Verilog settings. In that regime, close-but-broken edits pass the visible tests and fail held-out ones. The visible-tests baseline will measure this directly. If it matches differential execution on held-out success, our proposed fix is unnecessary and the contribution reduces to the diagnosis. We will report that outcome as such.
Why not just use a stronger LLM judge? SummExecEdit shows that a frontier model often blames the wrong span when it spots an error. Spurious Correlations shows that learned judges fall below a copy heuristic once copying is decorrelated from truth. Neither result is about code, though. The disagreement-set accuracy of a frontier judge is therefore a measured outcome here, not an assumption. If it is well above the smallest-edit heuristic, the lazy-judge hypothesis fails for strong models.
Why not just generate more unit tests? Generated tests need correct expected outputs, which the loop cannot obtain without an oracle. Differential execution sidesteps this by comparing versions rather than checking answers. It is still close to test generation, so we include a generated-test acceptance baseline with matched execution budget. The comparison shows whether "compare versions" adds anything beyond "more inputs".
Why not just use an AST-aware metric like CodeBLEU against a reference? Out of the BLEU found code-specific metrics no better than ChrF against developer ratings. CodeTransOcean found CodeBLEU nearly uncorrelated with execution. A reference also does not exist at repair time. We include CodeBLEU-to-predecessor only inside the diagnosis.
Proposed Experiments and Decisive Checks
The first check is the diagnosis. If MEIC-style scorers and LLM judges are lazy, ChrF-to-predecessor should predict their verdicts better than held-out correctness. Their accuracy on the disagreement set should also sit near or below the smallest-edit heuristic, mirroring the about-38%-versus-40.45% pattern from summarization. If neither holds, the central hypothesis is false.
The second check is the plateau. Plot held-out success against round for each acceptance rule. The prediction is that only differential execution keeps rising after round three, on both LLMTrans Python and the Verilog tasks.
The third check is attribution. Stratify solved problems by the edit distance between starting and final correct code. The predicted gain should concentrate in the far stratum. We also count how often each rule rolls back a revision later confirmed correct, which should drop sharply under differential execution.
As a secondary, more expensive check, if budget allows we run the comparison on a SWE-bench Lite subset. Here the in-loop reviewer is MAGIS's QA agent, whose reported resolution rate is 25.33%. This tests whether the effect survives repository-scale edits.
References
- Out of the BLEU: How should we assess quality of the Code Generation models?
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Spurious Correlations in Reference-Free Evaluation of Text Generation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits
- SPADE: Synthesizing Assertions for Large Language Model Pipelines
- LMUNIT: Fine-grained Evaluation with Natural Language Unit Tests