Priority-Scoped Horizons for Iterative Code Refinement

When a language model repairs or improves code over many rounds, it is chasing several goals of unequal importance at once. The code must import only libraries that actually exist, it must pass its tests, it should stay readable, and ideally it should run fast. Current refinement loops handle all of these with one rule: keep editing for a fixed number of rounds and accept whatever the model's latest attempt produces. The evidence says this rule fails in two ways. Most successful repairs happen on the first retry, and later rounds mostly stall (Rapid Fixes, Gradual Failures). Code produced by agents across evolving specifications grows more tangled and verbose far faster than human-written code (SlopCodeBench). A third failure is less visible: more aggressive exploratory search raises the rate of confident but unsupported code (Thompson Sampling Trees).

We propose to treat these goals as a strict priority order, and to give each priority level its own short "optimization horizon", meaning the number of edit attempts the loop may spend on that goal before it stops and reverts to the best version seen so far. Edits aimed at a lower goal, such as speed, are accepted only if every higher goal (no hallucinated dependencies, tests passing, no added structural complexity) is preserved within a small tolerance. Every check is performed by an external program, never by the model itself. The claim, in two sentences: refinement loops erode and hallucinate because a single, overly long horizon is applied to objectives whose feedback signals differ in reliability. Scoping the horizon per priority level, with a rollback rule, should keep the gains of iteration and remove most of the damage.

Core Claim

Each input direction plays a specific role. Iterative code refinement is the application regime. Hierarchical reward prioritization supplies the acceptance rule. The horizon tradeoff supplies the control principle. Hallucination is both a top-priority constraint and a key outcome measure.

Work on horizons in control and offline RL motivates the design. When the model used for lookahead is imperfect, longer rollouts compound its errors. Truncating the horizon then acts as a regularizer, as shown for Bayesian optimization rollouts (Why Non-myopic Bayesian Optimization is Promising), for offline imitation (Offline Imitation Learning by Controlling the Effective Planning Horizon), and for truncated Q-estimation (Offline Opponent Modeling). An LLM proposing code edits is exactly such an imperfect model of what its edits will do. The stagnation reported by Rapid Fixes, Gradual Failures is the code-refinement version of that delusion.

The quality of the feedback signal also differs sharply across goals. Package existence and test outcomes are nearly noise-free. Runtime is noisy enough that graded speed rewards break down (Making Execution Time a Trainable Reward), and strict monotone speed filtering overfits to the tests the loop has seen (ICER). Our hypothesis is that the right horizon shrinks as the signal gets noisier and the goal gets lower in priority. Lexicographic gating, meaning a lower goal only counts once every higher goal is met, is what makes a short horizon on a low tier safe. Under that rule, a wasted round on speed cannot cost correctness.

Formally, let $c$ be the current program. Let $f_0,\dots,f_K$ be externally computed scores in priority order: dependency and API grounding, held-in test pass rate, negated structural erosion and verbosity, and a thresholded runtime percentile bin. Let $\tau_j$ be the tolerance allowed at tier $j$. A candidate $c'$ proposed while working on tier $k$ is accepted when

$$ f_j(c') \ge f_j(c) - \tau_j \ \text{for all } j<k, \quad \text{and} \quad f_k(c') > f_k(c). $$

Tier $k$ is abandoned after $H_k$ consecutive rejected proposals. The loop then rolls back to the lexicographically best checkpoint, which is the rollback idea of Fang et al. made explicit. The tier-0 tolerance is zero, because a hallucinated package is never traded for anything. This acceptance rule is the value-based lexicographic filtering of Lexicographic Multi-Objective RL, moved from action selection to program selection.

Relation to the Closest Work

Making Execution Time a Trainable Reward is the closest prior work. It uses the same correctness-before-speed gate, with a binary percentile bonus that holds up under timing noise. However, it applies the gate during single-turn offline RL training, and it lists iterative repair as future work. We borrow its thresholded speed signal. We differ in applying the gate at test time across many rounds, in adding grounding and erosion tiers, and in controlling the horizon.

Textual Model Predictive Control regenerates text chunk by chunk against subgoals. It is the only work in the file that explicitly manages a horizon in LLM generation, but it optimizes a single reward within one response. We operate across rounds, over multiple ordered objectives. CodePRM localizes the weakest reasoning step and refines only that step. Its signal is test pass rate alone, and it offers no protection against regressions on non-functional properties.

ICER seems to contradict us, since it finds that relaxed acceptance beats strict improvement filtering for speed. We address this directly. The tolerance $\tau_j$ and the binned runtime tier are our version of relaxation, and one experiment below sweeps from strict to fully relaxed acceptance. SlopCodeBench and the package-hallucination study (Evaluating Inference-Time Defenses) provide measurement tools but propose no loop-level control. Spontaneous Reward Hacking constrains the design: it shows that a model grading its own edits with shared context inflates its own scores. For this reason no tier in our loop uses a model judge.

Research Program

We will build a test-time controller around an open code model. Qwen2.5-Coder-7B, the backbone CodePRM builds on, serves as the primary model, with a stronger API model for scale checks. The controller implements the acceptance rule, per-tier horizons, and rollback.

Tier 0 resolves every import and third-party symbol against package registries and installed-module introspection. It counts standard-library modules correctly, following the measurement correction in Evaluating Inference-Time Defenses. Tier 1 runs visible tests. The test suite is split so that acceptance decisions only ever see one part, and all reported correctness comes from the held-out part. Tier 2 computes SlopCodeBench's erosion and verbosity measures. Tier 3 times the code with repeated runs and bins the result into percentile thresholds against reference solutions.

Beyond the method, we will measure empirically how the best horizon $H_k$ for each tier depends on signal noise. We will inject controlled multiplicative timing noise, which follows the noise form reported by Making Execution Time a Trainable Reward. We will also vary the tolerance $\tau$. The prediction is that the best $H_3$ falls toward one or two attempts as noise grows, while the best $H_1$ stays short but nonzero, consistent with the first-attempt cliff.

We deliberately exclude any trained controller. If the fixed-rule version works, a learned horizon policy is a natural follow-up, but it should not be needed to establish the claim.

Baselines

The naive reference point is the standard unconstrained loop. The model sees execution feedback and rewrites freely for a fixed ten rounds, and the final attempt is kept. This is the protocol under which Rapid Fixes, Gradual Failures reports that full-history context hurt GPT models on MBPP+. We run it with and without history. On SlopCodeBench, its closest published analogue is the benchmark's agents: the best of them pass about 14.8% of checkpoints, and erosion increases in 77% of their trajectories (SlopCodeBench). We rerun this baseline with our backbone rather than comparing across models.

The strongest existing test-time competitor for efficiency is ICER's relaxed-acceptance refinement (ICER). For Gemini-2.5-Pro it reports normalized execution time falling from 1.89x to 1.03x of golden solutions. We reimplement ICER with our backbone, the same model-call budget, and the same visible tests. Beating it means matching its held-out runtime while showing lower erosion growth and lower hallucination rates, or improving held-out runtime at equal erosion.

For correctness-gated speed, the reference number is 27.7% at $p_{10}$ with the binary percentile reward on Qwen 2.5 7B (Making Execution Time a Trainable Reward). That result comes from a trained policy. We therefore report both our test-time loop applied to the untrained model and our loop applied on top of a correctness-RL checkpoint, and we frame that paper's number as a ceiling to approach rather than a head-to-head target. For functional repair, CodePRM's Generate-Verify-Refine loop raised APPS Competition Pass@1 from 16% to 29% (CodePRM). We run it under the same budget as a correctness-only refiner, which checks that our gating costs no functional accuracy.

Two ablations isolate the key ingredients. The first keeps lexicographic acceptance but uses one shared horizon of ten attempts for all tiers. This separates gating from horizon scoping. The second keeps per-tier horizons but replaces gating with a tuned weighted sum of the same four scores. Weights are chosen on a validation split with the same tuning budget we give $\tau$ and $H_k$. For absolute package-hallucination rates on our task mix there is no reported result found on Lacuna, so all hallucination comparisons are internal to runs we conduct.

Why not just…?

Why not just sample N independent solutions and keep the best? For single-file correctness this is a serious threat. Given the first-attempt cliff, restart-and-select may well match iterative repair on HumanEval or MBPP, although Textual Model Predictive Control reports beating best-of-N at fewer calls. For that reason we do not stake the claim on those benchmarks. The main setting is multi-checkpoint evolution and efficiency improvement, where state must carry forward and a fresh sample throws away an already-correct program. We still run "lexicographic best-of-N": the same tier checks, used only for selection among independent samples, at an equal number of model calls. If that baseline equals our loop on SlopCodeBench and Effibench, the conclusion is that ordered selection, not iteration, is what matters, and we will report it that way.

Why not just use a weighted sum of the objectives? Existing evidence already argues against it. A scalar speed term rewards programs that crash instantly (Making Execution Time a Trainable Reward). Scalar driving rewards left collision rates at 32.9%, against 3.6% under thresholded lexicographic control (Urban Driving with Multi-Objective Deep RL). In a test-time loop with a tuned sum, however, the gap might be small. The weighted-sum ablation settles this, and it gets a tuning budget equal to ours.

Why not just cap refinement at one or two rounds, or prompt for clean code? A fixed short cap is the natural reading of the first-attempt cliff, and it is our closest simple rival, so we include caps of one and two rounds as baselines. Our loop should beat them only when lower-tier goals benefit from a few extra gated rounds. Quality prompts reduced initial slop in SlopCodeBench but did not slow its accumulation, and they slightly lowered correctness. We still rerun that prompt condition on our backbone.

Why not just use guided decoding or retrieval against package hallucination? According to Evaluating Inference-Time Defenses, decoding-only defenses collapse under adversarial prompts, with hallucination rates rising by up to 45 points, and retrieval helped in only 18 of 32 configurations. A registry check used as a hard gate is cheaper and does not rely on the model's own knowledge. We test this under that paper's adversarial prompts.

Proposed Experiments

The first decisive check is on SlopCodeBench. At an equal model-call budget, it compares the rate at which erosion and verbosity grow across checkpoints, along with the checkpoint pass rate, for the naive loop, the fixed short cap, lexicographic best-of-N, ICER, and our controller. Success means a clearly flatter erosion slope with no loss in checkpoints passed.

The second check runs on Effibench and the CodeContests-derived speed benchmark. It measures held-out-test runtime and $p_{10}$ while sweeping $\tau$ from strict to fully relaxed, which tests directly whether a tolerance-plus-binning rule reconciles ICER's finding with correctness gating.

The third check targets the hallucination tension reported for Thompson Sampling Trees. We raise exploration by increasing sampling temperature and the number of branches, and record the package hallucination rate and package utility under benign and adversarial prompts, with and without the tier-0 gate. The prediction is that exploration still raises correctness, while final-program hallucination stays flat only when the gate is on.

The fourth check is the timing-noise sweep over $H_3$. If the best horizon does not shrink with noise, the priority-scoped horizon idea is wrong even if the method's overall numbers look good.

On MBPP+ and APPS, we report only that correctness is not harmed. The reference points there are TMPC's 61% on MBPP and CodePRM's 29% on APPS Competition, each measured under its own backbone.