Execution-Checked Confidence for Code Self-Repair

When a language model fixes its own buggy program, it first writes a diagnosis ("the loop skips the last element because the range stops at n-1") and then edits the code to match. Prior studies found this loop barely beats, and often loses to, simply sampling fresh programs from scratch at the same cost. The cause is that roughly 40% of the model's diagnoses are fabricated, and the loop has no way to notice. Diagnoses about code differ from most hallucinations, though: much of what they assert can be checked mechanically, by running the program and looking. We propose to make every diagnosis carry small executable predictions about the failing program's actual behavior. We then train the model's stated confidence in a fix against outcomes the interpreter reports for free, and use that calibrated confidence to decide, at each turn, whether to repair, start over, or stop.

The bet is that calibration, which earlier work demonstrated only for single-pass question answering and math, is the missing control signal in iterative code refinement. If the model knows when its diagnosis is wrong, a repair loop should never do worse than resampling at equal compute. It should do better on hard problems, where fresh samples almost never succeed.

Core Claim

Self-repair fails because the step that produces feedback is uncalibrated, not because repair is useless. The two works that formalized the repair loop as a tree of generate, execute, explain, and repair stages (Demystifying GPT Self-Repair for Code Generation; Is Self-Repair a Silver Bullet for Code Generation?) motivate this claim. They found that GPT-4's explanations were obviously inaccurate in 32 of 80 audited cases. Self-repair gained only about 5–8% over compute-matched independent sampling on APPS, and it lost to sampling for weaker models. Swapping in human feedback raised repair success from 33.3% to 52.6%. Accurate diagnosis is therefore worth a great deal, and its absence is what makes the loop costly.

Our claim has two parts. First, a diagnosis can be broken into a checkable part and an uncheckable part. The checkable part is "on the failing input, variable j equals 4 at line 12." The uncheckable part is "it should equal 5." The checkable part can be verified by instrumented execution at test time. Second, the model's confidence that its proposed repair will succeed can be trained with a proper scoring rule, a reward that is maximized only by reporting honest probabilities, using execution outcomes as labels. A loop that routes on this confidence should weakly dominate both naive self-repair and independent resampling.

Relation to Prior Work

The training method comes from Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. That paper adds a Brier-score penalty on verbalized confidence to correctness rewards, a method it calls RLCR. On HotpotQA it cut expected calibration error from 0.37 to 0.03, and it outperformed external post-hoc classifiers. It was evaluated only on single-pass QA and math. We move the confidence target from a final answer to a diagnosis-plus-fix inside a multi-turn loop, and we add a second, denser label source: the truth value of the diagnosis's executable probes.

Two papers constrain the design. Trace Length is a Simple Uncertainty Signal in Reasoning Models shows that reasoning length is a strong zero-shot uncertainty signal, and that combining it with verbalized confidence wins in 110 of 120 settings. The same paper reports that the signal degrades below 40% accuracy, which is exactly where weak models sit on APPS. We therefore treat trace length as a baseline to beat rather than as our mechanism. Inference-Time Conformal Reasoning with Valid Factuality Control for Large Language Models bounds the validity of intermediate reasoning claims and halts generation when they fail. It cannot repair, and it uses no interpreter. Our router keeps its "stop when unreliable" idea and adds "restart" and "repair" as options.

Three papers motivate the use of execution-checkable probes. CRUXEval shows that models which generate code well often cannot predict what code does (Code Llama 34B scores 46% on output prediction). This suggests fabricated diagnoses are often wrong about current behavior, which is precisely what probes check. Code Hallucination documents "logic blindness," where models assert that seeded-bug code is correct. Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics shows that passing outputs can hide unsound logic, a warning that test pass/fail alone is a noisy label.

The closest system is ExeCRE. ExeCRE estimates the reliability of candidate reference programs from their agreement on about 1,000 generated inputs, using Dawid-Skene expectation-maximization. It ignores the model's internal confidence and is fooled when candidates share a flaw. Our method checks the model's own claims about one program and calibrates its confidence. The two are complementary, and we compare against ExeCRE directly. OpenCodeInterpreter supplies the training data and a base model. It trusts execution feedback and its own explanations without any confidence estimate.

Research Program

At each repair turn, the model receives the program, the failing test, and the error output. It produces three things. The first is a free-text diagnosis. The second is one to three probes: assertions about intermediate values at specific lines on the failing input. Each probe carries its own confidence, and each must reference a line the subsequent patch edits. The third is a patch with a confidence $q \in [0,1]$ that the patched program passes the visible failing test without breaking previously passing ones.

An instrumented interpreter then evaluates each probe, which yields a truth label $z_i \in \{0,1\}$ for probe $i$. Running the patch yields a success label $s \in \{0,1\}$. Training maximizes

$$R = s - (q - s)^2 + \lambda \sum_i \left[ -(p_i - z_i)^2 \right],$$

where $p_i$ is the stated confidence in probe $i$ and $\lambda$ weights probe calibration. We use the bounded Brier form because the RLCR paper reports that log-loss rewards collapse into refusal. Probes that restate the program text, such as checking a constant on a line that assigns it, are detected by a simple static check and earn zero probe reward. This removes the cheapest way to game the probe term.

At inference, a router reads $q$ along with the probe outcomes. Any falsified probe caps $q$. The router repairs if $q$ exceeds a threshold $\tau$, otherwise discards the lineage and draws a fresh sample, and stops once a candidate passes all visible tests. The threshold $\tau$ is set on a held-out split for each compute budget. If $q$ is calibrated, it is directly comparable to the base rate at which fresh samples succeed, which is why a single threshold suffices.

Training data comes from APPS training problems and the Code-Feedback multi-turn sessions released with OpenCodeInterpreter. We use Code Llama-13B, where self-repair currently loses to sampling, an OpenCodeInterpreter checkpoint, and one open reasoning model (an R1-distilled or OpenThinker2 model) so that the trace-length baseline is meaningful.

Baselines

The naive reference point is compute-matched independent sampling. We draw programs at temperature above zero, keep the first that passes visible tests, and report pass rate as a function of total generated tokens. This is the curve both self-repair papers showed is hard to beat. On APPS the strongest reported self-repair result is GPT-4 beating this baseline by about 5–8% (Is Self-Repair a Silver Bullet for Code Generation?). For weaker models, no positive margin was reported. Beating independent sampling at every budget for Code Llama-13B would already reverse a published negative result.

The second baseline is uncalibrated self-repair, built from the repair-tree procedure of those papers and run on our models with the same token budgets. A third family covers zero-shot confidence signals: prompted verbalized confidence, trace length, and their combination from the trace-length paper. Each is plugged into the same router, with the same threshold-tuning protocol.

The strongest existing method is ExeCRE. It reports raising GPT-5.2 on LiveCodeBench v6 from 62.7% to 72.8% Pass@1, and cutting unnecessary revisions of already-correct code from 113.2 to 14.0 (ExeCRE). We run ExeCRE on the same open models and problems, counting its input generation and executions against the compute budget. To claim a win, our method must match its pass rate at lower cost or exceed it at equal cost. It must also cut correct-code revisions by a comparable factor.

The ablations isolate the key ingredient in three ways. Probe checking at test time without calibration training tests whether probes alone suffice. Calibration training on $s$ only ($\lambda = 0$) tests whether the probe labels add anything. An external classifier, trained on the same execution labels to predict $s$ from the transcript, tests whether internal confidence matters. All variants share data, base model, and RL compute.

Why not just…?

Why not just resample independently? The router contains resampling as an action, so in the worst case it should match independent sampling. The open question is whether it ever beats it. The self-repair papers note that repair has its best chance on competition-level problems, where fresh samples almost never succeed. We report APPS results by difficulty level. If gains appear only where diagnoses are accurate, that supports the claim that feedback quality is the bottleneck.

Why not just ask the model how confident it is? RLCR found that confidence after binary-reward training is badly miscalibrated (ECE 0.37). The trace-length paper finds that prompted confidence is wording-sensitive and that trace length weakens below 40% accuracy. This alternative is cheap, so we run it as a full baseline inside the identical router. If it matches the trained router, the training component is unnecessary and we will say so.

Why not just check the probes at test time and skip training? This is the real threat, since probe checking catches diagnoses that misdescribe current behavior with no training at all. It cannot assess the "should be" half of a diagnosis or the patch itself, which is what $q$ is trained for. The probe-only ablation settles this. If it recovers most of the gain, the contribution narrows to a training-free probe protocol, and that is still a useful finding.

Why not just use ExeCRE? ExeCRE answers a different question, namely which reference programs or tests to trust. It costs about 26 seconds per problem and fails on shared algorithmic flaws. We test it head-to-head and also combined with our router. Where all sampled candidates share a bug, ExeCRE's agreement signal cannot detect it, but a falsified probe can.

Proposed Experiments

The primary experiment plots pass rate against total tokens on APPS (by difficulty), HumanEval, and LiveCodeBench v6 for every baseline. For reference, OpenCodeInterpreter-33B's HumanEval Pass@1 rises from 79.3 to 82.9 with execution feedback in the OpenCodeInterpreter paper.

The calibration experiment reports, at the diagnosis level, expected calibration error and AUROC of $q$ and $p_i$ against execution truth. It also re-runs the self-repair papers' audit protocol: humans label 80 sampled diagnoses as fabricated or not, and we measure how many fabricated ones the router rejects.

To check transfer of probe accuracy, we evaluate the probe-calibrated model on CRUXEval output prediction. Improving there would indicate the probe reward teaches mental execution, rather than only making the model hedge.

Three decisive checks follow. The first asks whether the router beats compute-matched independent sampling for Code Llama-13B on APPS, where naive self-repair currently loses. The second asks whether the full method beats both the probe-only and the $\lambda = 0$ ablations. The third asks whether calibration holds on LiveCodeBench problems released after the training data, given that out-of-distribution miscalibration is the known weakness of calibration training. A failure on the first check would falsify the core claim. A failure on the second or third would narrow the claim to whichever component survives.