When to Stop Debugging: Recall-Based Stopping Rules for LLM Code Refinement

Language models that write code are now usually run in a loop. They write a program, run it, read the errors or test failures, and try again. Nobody has a good answer to when that loop should end. Current systems stop after a fixed number of rounds, stop the moment visible tests pass, or stop when two reviewing agents agree. The evidence says all three choices are poor. Gains flatten after one or two rounds. Refining until the visible tests pass makes patches more likely to game those tests and fail the hidden ones. Code also gets more bloated and tangled with each extra round. Optimal stopping theory, the mathematics of deciding whether to act now or keep searching, offers a precise alternative. Under a condition it calls monotonicity, the best policy reduces to a simple threshold: stop as soon as the expected gain from one more attempt falls below its cost.

This note proposes treating each refinement loop as a search problem in which the program can go back to an earlier attempt, called "recall." The loop keeps every attempt it has made. After each round, a calibrated predictor estimates two things: how likely the best attempt so far is to pass the hidden acceptance tests, and how much one more round would raise that likelihood. The loop stops when that expected gain drops below a single cost parameter, and it returns the best attempt so far rather than the latest one. What is new is the quantity being thresholded. It is not a reward-model score or a visible-test pass. It is a hidden-test-grounded estimate of the marginal gain of one more round, applied to a best-so-far pool. The proposal also tests empirically whether real refinement trajectories satisfy the monotonicity condition that makes this greedy rule optimal.

Core Claim

Let $h_t$ be everything observed after round $t$: the candidates, their tracebacks, visible-test results, diff sizes and complexity measures. Let $Q_i$ be the probability that candidate $i$ passes the hidden tests, predicted from $h_t$, and let $B_t = \max_{i \le t} Q_i$ be the value of the best candidate kept so far. The proposed rule stops at the first round where

$$\mathbb{E}\left[\max(B_t, Q_{t+1}) - B_t \mid h_t\right] \le c,$$

and returns the best-so-far candidate. Here $c$ is the price of one more round, expressed in tokens and execution time converted into units of hidden-pass probability. This is the one-step-look-ahead rule described by Chaporkar et al.. If the expected one-step gain is non-increasing along a trajectory (the "monotone case"), stopping is absorbing, in the sense of Nutz: once stopping is optimal it stays optimal, and the one-step rule matches the full look-ahead policy. Sweeping $c$ traces the whole cost–quality curve with one knob.

The claim has three parts. First, at matched token budgets, this rule beats fixed-round truncation with keep-best selection on hidden-test pass rate. Second, it lowers the test-overfitting rate relative to refining until visible tests pass. Third, whether the monotone case holds can be measured on real trajectories, and it predicts where the greedy rule fails.

Relation to Prior Work

Two papers supply the method. T-TAMER frames model-serving cascades as a "Pandora's box" search, in which each option costs something to open and the searcher may keep any opened option. It proves that strategies forced to accept the most recent model's output cannot reach a constant-factor approximation of the optimal trade-off, while strategies with recall can. T-TAMER works on static classification, with known transition statistics and only an accuracy–latency objective. We carry its recall result into generative refinement, where each "box" is a new program produced conditionally on earlier failures, and the reward is unobserved hidden-test correctness.

Martingale Foresight Sampling uses martingale convergence to stop foresight rollouts in math reasoning. It never executes code, and its stopping signal comes from the model's own probabilities. Our signal comes from execution feedback, calibrated against hidden tests.

Two papers come closest in spirit. DREAM+ stops sampling when a process reward model, a learned scorer of intermediate steps, crosses an upper threshold, and it moves budget to steps that fall below a lower threshold. Its thresholds are tuned cutoffs on text-only scores, with no execution signal and no recall over earlier candidates. ICER shows that strictly accepting only improving versions overfits to timing jitter. It therefore accepts any passing candidate and stops after a small fixed number of rounds. Our recall pool resolves the tension between ICER's relaxed acceptance and MEIC's rollbacks to the last good state. The loop can explore freely because it can always fall back on the best candidate it has kept.

Three results motivate and constrain the design. Investigating Test Overfitting on SWE-bench finds that refining against generated reproduction tests raises overfitting from 21.8% to 25.5% for Claude-3.7 Sonnet and from 33.0% to 35.9% for GPT-4o. Hiding test contents does not fix this. That finding rules out any stopping signal built on visible-test passing alone. SlopCodeBench finds structural erosion growing in 77% of trajectories and verbosity growing in 75.5%. So continuing has a cost beyond tokens, and erosion features belong in $h_t$. CodeTransOcean reports that self-debugging success rises from 48.57% with no debugging rounds to 52.57% after three. The marginal gain shrinks quickly, which is the shape the monotone case needs.

Adversarial Review stops on agreement between reviewing agents, at about 4.5 times zero-shot cost. Its authors list adaptive, risk-triggered review as open. Our rule is one concrete form of that idea.

Research Program

The work has four parts.

The first is a trajectory corpus. On training splits we run long refinement loops of up to eight rounds, with two open-weight models and one frontier model. We record every candidate and label each one with its hidden-test outcome. On LiveCodeBench, the hidden tests are the held-out judge tests. On the LLMTrans subset of CodeTransOcean, the label is execution success. On a SWE-bench Verified subset, the label is the full acceptance suite. We also record visible-test results, tracebacks, diff size and the complexity metrics used by SlopCodeBench.

The second part is a calibrated value model. A small model maps $h_t$ to two outputs: $Q_t$, and the distribution of $Q_{t+1}$ given $h_t$, which is what the expected one-step gain needs. We check calibration with reliability curves on held-out problems. Calibration matters because the threshold only means something if the probabilities are honest.

The third part is the stopping rule with recall described above. We compare it against variants that drop recall, and against variants that threshold visible-test signals instead of predicted hidden correctness.

The fourth part is a monotonicity audit. For each trajectory we test whether the estimated one-step gain is non-increasing. We report the fraction of trajectories in the monotone case, and we compare the one-step rule against a two-step look-ahead on the trajectories that are not monotone. ICER's data-structure switches, which look neutral before they pay off, are the expected counterexamples.

Distribution shift is a known weakness of calibrated stopping and is flagged explicitly in T-TAMER. We probe it by calibrating on one benchmark or model and testing on another.

Baselines

The naive reference point is a single shot with no refinement. It is the lowest-cost corner of every cost–quality plot. Adversarial Review reports 77% zero-shot pass rate on LiveCodeBench and 71.6% on SWE-bench Verified.

The strongest simple competitor is fixed-round truncation with keep-best. The loop runs $K$ rounds for $K$ from 1 to 8, and returns the candidate with the best visible-test score, breaking ties by most recent. It uses recall and exploits the fact that ICER converges in one or two rounds, so it is the real threat. For it, no reported result was found on Lacuna. The closest figure is CodeTransOcean's 52.57% after three rounds, which returns the last candidate rather than the best. We run it on the same trajectories, so every policy gets the same samples. We compare full cost–quality curves, and the proposal wins only if its curve lies above this one with confidence intervals that do not overlap at some budget.

Refine-until-visible-pass is the default in most agents and the target of the overfitting finding. Beating it means a lower hidden-test overfitting rate at equal or higher resolve rate.

A tuned score threshold in the style of DREAM+ applies two thresholds to the value model's raw $Q_t$, grid-searched on the same calibration split with the same number of configurations. It isolates whether the stopping theory adds anything beyond "a good scorer plus a tuned cutoff."

The strongest published system is Adversarial Review. It reports 87% on LiveCodeBench and 75.2% on SWE-bench Verified, at about 4.5 times zero-shot tokens. We do not aim to beat its pass rate at lower cost with a weaker generator. We aim to match it at a fraction of its tokens when the generator is held equal, or to raise its pass rate per token when our rule decides when its review loop runs.

Two ablations isolate the key ingredients. The no-recall ablation returns the latest candidate under the same rule, and tests whether T-TAMER's separation shows up in code. The visible-target ablation trains the value model on visible-test outcomes instead of hidden ones, and tests whether hidden-test grounding is what reduces overfitting.

Why not just…?

Why not just run two rounds and keep the best? ICER's convergence within one or two rounds and the CodeTransOcean plateau suggest that a small fixed $K$ with keep-best captures most of the gain. This is the main threat, and we accept that it may win on easy benchmarks. The rule can only help through heterogeneity. Some problems are finished at round zero, as DREAM+ found for about 80% of GSM8K steps. Others keep improving past round two. The trajectory corpus measures this directly. If the variance across problems of the round at which hidden-pass peaks is small, the adaptive rule cannot beat the best fixed $K$, and we will report that. We expect a real gap on SWE-bench Verified and LiveCodeBench hard problems rather than on LLMTrans.

Why not just stop when the visible tests pass? The SWE-bench overfitting study shows this target is gamed: overfitting rises under refinement, and hiding test contents leaves it high. The visible-target ablation and the overfitting-rate comparison test whether thresholding predicted hidden correctness does better. If hidden-test grounding gives no reduction, the core claim fails.

Why not just tune a cutoff on a learned scorer, as DREAM+ does? With a good scorer, a tuned cutoff might match the derived rule. The tuned-threshold baseline uses the same value model, so any gap comes from the decision rule. That rule has two parts: recall, and a marginal-gain criterion in place of a level criterion. We expect the derived rule to generalize better across budgets and across shifted benchmarks, because one parameter $c$ has a cost meaning, whereas tuned levels do not carry over. A held-out-benchmark comparison settles this.

Why not just add more reviewing agents? Adversarial Review already shows a five-agent debate doing worse than its three-agent protocol, 72.6% against 75.2% on SWE-bench Verified, with large token costs. Our rule is complementary rather than a substitute: we test it as the trigger that decides when that review loop runs.

Proposed Experiments and Decisive Checks

The main experiment plots hidden-test pass rate against total tokens on LiveCodeBench, LLMTrans and a SWE-bench Verified subset, for all policies over a shared trajectory corpus. Secondary metrics are the overfitting rate on SWE-bench Verified and the erosion and verbosity growth on SlopCodeBench checkpoints, measured at each policy's stopping point. We also measure Effibench runtime relative to the golden solution, with ICER's 1.89 to 1.03 times as reference.

Three results are decisive.

The first decides whether the method is worth anything. The rule must lie above fixed-$K$ keep-best on at least the two harder benchmarks. If it does not, the conclusion is that refinement trajectories are too uniform for adaptive stopping to pay off, and we report that.

The second decides whether the theory is useful. The no-recall ablation should lose clearly, and the monotone fraction should predict where the one-step rule matches two-step look-ahead. If the monotone case is rare, greedy thresholding is not justified for code, and the result becomes an argument for look-ahead.

The third ties the method to the known failure modes. Hidden-target stopping should cut overfitting and erosion relative to refine-to-pass at equal resolve rate.