Code-repair systems built on large language models work in a loop. They write a patch, run tests, read the failures, and write another patch. They stop after a fixed number of rounds or when the tests pass, and then they submit whatever they have. Two findings make this risky. Gains flatten after a round or two: on CodeTransOcean, three rounds of compiler feedback raised ChatGPT's debugging success from 48.57% to only 52.57% (CodeTransOcean). Later rounds can also make things worse, because models repeatedly prompted on failure logs corrupt surrounding code or introduce vulnerabilities (Srivastava et al.). What a deployed system needs is a stopping rule. After each round it should choose between three actions: accept this patch, try another round, or hand the issue to a human. It should also come with a statistical guarantee that no more than a chosen fraction, say 10% or 20%, of accepted patches are wrong.
We propose such a rule and pair it with a specific confidence signal. The signal is a probability that the patch is correct, computed by keeping a belief over which functions in the repository might now misbehave. Beliefs are linked along call and import edges, and each test outcome counts as evidence about the functions that test executes. Each refinement round edits only a few functions and reruns only a few tests. The belief can therefore be updated incrementally, recomputing only along the graph paths the edit touches. The same machinery decides which tests are worth rerunning at all. The claim to be tested is that this structural signal catches the most dangerous failure mode better than a flat confidence score: a patch that passes its own reproduction test but silently breaks a caller. The claim also requires that the stopping rule keep its error budget on held-out issues.
Core Claim
The work makes two separable contributions, and the experiments are designed so that one can succeed without the other.
The first is a risk-controlled accept/refine/defer policy over whole refinement trajectories. The policy has a small parameter vector $\lambda$: a per-round acceptance threshold on the confidence score, and a "futility" threshold. When the score fails to improve by more than the futility threshold between rounds, the system defers. For a fixed $\lambda$, running the policy on one issue yields a single outcome: either a deferral, or an accepted patch that is right or wrong against the hidden tests. Because the policy is scored once per issue, repeatedly peeking at the score across rounds does not break the guarantee. We choose $\lambda$ on a calibration split so that
$$\Pr\big(\text{accepted patch fails hidden tests} \mid \text{accepted}\big) \le \epsilon \quad \text{with probability } 1-\delta,$$
where $\epsilon$ is the user's error budget and $\delta$ is the allowed chance that the calibration itself is misleading. The bound uses the binomial-inversion test of Selective Classification for Deep Neural Networks, which calibrates a threshold on softmax confidence for a fixed classifier. Because we search over many candidate $\lambda$, the tests are ordered from most to least conservative and applied in that order, so the guarantee still holds after the search. Among the $\lambda$ that pass, we pick the one with the highest acceptance rate (coverage).
This extends Selective Code Generation for Functional Guarantees. SCG bounds the false discovery rate of generated functions against fuzzed test suites, but it is a one-shot filter: it abstains and stops, and it needs a reference solution to seed its fuzzer. Our policy acts inside the loop. It needs no reference solution at inference time, and its loss is defined by the benchmark's hidden tests, used only during calibration.
The second contribution is the confidence score itself. We build a dependency graph of the repository in the style of SynFix, whose RelationGraph links files, classes and functions through calls, imports and inheritance. SynFix uses that graph only to prompt the model about the immediate neighbors of an edited node. We instead place a binary variable on each function near an edit, meaning "this function still behaves as intended." Three kinds of factors connect these variables:
- Edit priors. Each edited hunk gets a prior probability of being faulty from a learned self-evaluation score, in the manner of ASPIRE. ASPIRE trains a soft prompt to judge the model's own answers and showed that this beats prompted "P(True)" and perplexity on question answering.
- Coupling factors. A possibly faulty callee raises the fault probability of its callers, following call edges.
- Test factors. Each test is tied to the functions it executes, as measured by one instrumented coverage run.
The score is the posterior probability that every edited or affected function is fine, given all test outcomes observed so far.
Research Program
The inference machinery is borrowed directly from incremental message passing. We restrict the graph to functions within a few call hops of any edit. Where the restricted graph has small treewidth, we compile it into a junction tree (Andersen et al.). A junction tree groups variables into clusters arranged as a tree, so that passing messages between clusters gives exact marginals. When a round edits one function, only the messages on the tree path from that function's cluster need recomputing. This is the path-restriction result of Yuan & Hansen, which they established for branch-and-bound MAP search. When a rerun test returns the same outcome as before, propagation stops early, in the way C3 halts re-execution once a callsite's value matches its cache. Where the treewidth is too large, which is a known weakness of junction trees, we fall back to loopy belief propagation. We report how often this happens.
We should be honest about where incremental computation matters. Belief arithmetic over a few hundred binary nodes costs almost nothing next to a language-model call, so recomputing from scratch would be fine for the score alone. Incrementality pays off in test execution, which in repositories like django or sympy takes real wall-clock time. After an edit, only tests covering nodes whose belief changed by more than a tolerance $\tau$ are rerun. This mirrors the delta-threshold recomputation of Triebel et al.. The test-selection step is part of the scoring procedure, and the whole procedure is calibrated end to end, so skipping tests cannot quietly weaken the guarantee.
The pieces are trained and calibrated on separate data:
- Base refinement loop. Agentless-style localization, Search/Replace patching, and its own reproduction plus regression tests (Agentless), run for up to $K=5$ rounds.
- Learned components. Edit-prior prompts and coupling weights are fit on trajectories generated from SWE-bench-train (SWE-bench).
- Calibration and test. Disjoint splits of SWE-bench Verified and Lite.
We also report a split in which calibration and test issues come from different repositories. The guarantee assumes calibration and test issues are exchangeable, and the file already flags threshold drift under domain shift as an open problem. We would rather measure how badly the bound is violated under that shift than assume it holds.
Baselines
The naive reference point is "refine to $K$ and accept everything." It has 100% coverage, and its risk equals one minus the resolve rate. Agentless reports 32.00% (96/300) resolved on SWE-bench Lite at $0.70 per issue. SynFix reports 52.33% on Lite and 55.8% on Verified. We re-run the Agentless loop with the same backbone model, so every selective method in our comparison shares an identical trajectory generator. Any method beats this reference only if it lowers error among accepted patches at useful coverage.
The strongest practical competitor is the test-pass filter: accept a patch if and only if its self-generated reproduction test and the regression suite pass. This is exactly what Agentless and SynFix already do, it is cheap, and it is a real threat. It uses the same trajectories and test budget. To beat it, our policy must reach lower hidden-test error at equal coverage, or higher coverage at equal error.
The strongest existing selective method is SCG's approach: calibrate one threshold on the final-round confidence (Selective Code Generation for Functional Guarantees). We adapt it to SWE-bench by using hidden tests as the calibration loss in place of fuzzed tests, and give it the same calibration split. SCG reports keeping its false discovery rate below a budget of 0.3 on FuzzEval with GPT-4o and DeepSeek R1. No reported coverage number was found on Lacuna for SCG, and no reported result was found on Lacuna for any selective method on SWE-bench. We therefore also run our method on FuzzEval's HumanEval-f and MBPP-f, where at round zero it reduces to SCG. This lets us check that our re-implementation reproduces SCG's risk control before comparing further.
The key ablation replaces the graph posterior with a flat learned scorer under the identical stopping rule and calibration. The flat scorer is an ASPIRE-style self-evaluation plus trajectory features: tests passed, diff size, number of files touched, and round index. This isolates whether structure adds anything. A second ablation replaces the stopping rule with a fixed round count. A third replaces incremental test selection with full reruns, measuring only the cost difference.
Why not just…?
Why not just accept when the tests pass? The reproduction test is written by the same model that wrote the patch. Regression suites also rarely exercise every caller of an edited function. So "tests pass" can coincide with a hidden regression in untested code, which is the failure Srivastava et al. describe. We will measure directly how many test-passing patches fail the hidden PASS_TO_PASS tests. If that fraction is already below the target $\epsilon$, the test-pass filter is sufficient, and we will say so.
Why not just calibrate SCG's threshold on the last round? That approach still spends all $K$ rounds on hopeless issues. It also cannot recognize that round 2 was better than round 4. The plateau in CodeTransOcean suggests most of those later rounds are wasted. At a fixed $\epsilon$, it may reach coverage similar to ours. The comparison therefore reports tokens and test minutes per issue alongside coverage, and it counts how often the final patch is worse than an earlier one.
Why not just a learned classifier on flat trajectory features? This is the most serious threat, because a classifier trained on the same data might learn "large diffs to heavily-called functions are risky" without any graph. The flat-scorer ablation settles it. If the graph posterior does not improve coverage at fixed $\epsilon$, specifically on issues whose patches touch functions with many callers, we drop the graph from the score. We would then keep the risk-controlled stopping rule and the graph-based test selection as the contributions.
Proposed Experiments
The decisive check is realized risk on held-out issues. For $\epsilon \in \{0.1, 0.2, 0.3\}$ and $\delta = 0.05$, the fraction of accepted patches that fail hidden tests must stay at or below $\epsilon$. This must hold across repeated random calibration/test splits of SWE-bench Verified, and we separately report violations under the cross-repository split.
The main comparisons are coverage at each $\epsilon$ against the three baselines above, plus language-model tokens and test-execution minutes per issue. We also report a regression-specific metric: among errors that each method accepts, the share that are PASS_TO_PASS failures. The graph score is supposed to reduce exactly that share.
A secondary check on CodeTransOcean's LLMTrans tests whether the futility rule recovers the DSR@3 success rate while spending fewer rounds, given that gains there plateau after round 0. A structural diagnostic reports treewidth of the restricted graphs and the fraction of issues that need the loopy fallback.
The proposal fails if risk control breaks on held-out data. It is reduced to its stopping-rule half if the flat scorer matches the graph score.
References
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Do Users Write More Insecure Code with AI Assistants?
- Selective Classification for Deep Neural Networks
- Selective Code Generation for Functional Guarantees
- SynFix: Dependency-Aware Program Repair via RelationGraph Analysis
- Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
- HUGIN - A Shell for Building Bayesian Belief Universes for Expert Systems
- Efficient Computation of Jointree Bounds for Systematic MAP Search
- C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching
- Parsing Outdoor Scenes from Streamed 3D Laser Data Using Online Clustering and Incremental Belief Updates
- Agentless: Demystifying LLM-based Software Engineering Agents
- SWE-bench: Can Language Models Resolve Real-world Github Issues?