Widen or Deepen: Execution-Signature Diversity as a Test-Time Budget Rule for Code Repair

Coding agents that fix bugs have two ways to spend extra compute. They can go deeper, feeding test failures back to the same attempt for another round of edits. Or they can go wider, starting more independent attempts, possibly from different models, and picking among them. Both stop paying off quickly. Compiler-feedback self-debugging gains little after two or three rounds (CodeTransOcean), and efficiency refinement converges within one or two (ICER). Adding agents to a debate beyond three also hurts (Adversarial Review). The ensemble literature gives one explanation for both plateaus: extra members only help when their errors are uncorrelated (Dynamic Optimizations of LLM Ensembles). We propose that this is also why extra refinement rounds stop helping. When a model is asked to revise its own work, it tends to repeat its earlier mistakes, so successive rounds make correlated errors.

The proposal is a per-problem, training-free rule that decides at each step whether to deepen, widen, or stop. The rule reads only what the sandbox reports: which tests fail, what exception is raised, and where in the code it is raised. We call that tuple a candidate's execution signature. If a refinement round changes the signature, the attempt is making progress, so we keep deepening it. If the signature stops changing, and other attempts are failing with the same signature, the ensemble's errors have become correlated. We then widen, but only with a member drawn to be different: another model, another scaffold, or a fresh context. When the passing candidates agree in behavior, we stop. The claim is that this rule reaches the accuracy of fixed best-of-16 sampling or a fixed three-agent review at a fraction of the tokens. A secondary claim is that it slows the code erosion that long single-trajectory refinement causes (SlopCodeBench).

Core Claim

Consider the failing candidates at a given step, $F$, and the set of distinct execution signatures among them. Their co-failure concentration is

$$c = 1 - \frac{|\text{distinct signatures in } F|}{|F|}.$$

This number is near 0 when every candidate fails differently and near 1 when they all fail the same way. It is a black-box stand-in for the focal diversity of Dynamic Optimizations of LLM Ensembles. That work measures how often models fail together using output probabilities, on multiple-choice tasks, and learns a reinforcement-learning pruner that must be pretrained on historical queries. We measure the same quantity from execution outcomes. That makes it usable with API-only models, and it requires no warm start.

The hypothesis has two parts. The first is that $c$, together with whether an attempt's own signature is stalled, predicts whether the next unit of compute gives more resolved problems when spent on depth or on breadth. The second is that allocating compute this way dominates any fixed allocation on the curve of accuracy against tokens.

Relation to Prior Work

Four lines of prior work bear on the proposal, each in a different role.

Evidence that fixed sizes waste compute (motivation). Adversarial Review fixes a Main agent, a Reviewer, and a Critic for every problem. It beats a five-agent system, costs about 4.5 times the tokens of zero-shot generation, and names adaptive triggering as future work. SWE-Gym samples a fixed $k=16$ trajectories per problem and ranks them with a learned verifier. It reaches 32.0% Best@16 while the oracle ceiling (Pass@16) is 42.8%. Both show the cost of fixed sizes. Neither conditions its size on what execution reveals.

Evidence that constrains the design. Multi-teacher distillation peaks at a few teachers because averaging washes out sharp signals (Class Incremental Learning with Multi-Teacher Distillation). Pseudo-labeling degrades past three models (When CNN Meet with ViT). Together these warn against widening by adding more copies of the same model. So our widening step must inject diversity, not just more samples. ICER shows that strict monotone acceptance overfits to the visible tests. We therefore accept any candidate that passes the visible tests, rather than requiring each step to improve on the last. MEIC's rollback to the last non-regressing version (MEIC) is reused as the deepening step's safety net.

Methods we reuse. The structure of the signature follows BOOSTAPR, which maps stack traces to the edit lines that caused them. We use only the location, not the learned credit model.

The erosion lens (secondary claim). SlopCodeBench shows structural erosion rising in 77% of agent trajectories. Our secondary hypothesis follows from this: fresh wide starts reset the accumulated bloat that long deep trajectories build up.

Research Program

The first step is measurement, not method. On SWE-Gym Lite (230 instances), we run a dense grid over depth (1 to 6 rounds) and breadth (1 to 16 attempts) with Qwen-2.5-Coder-32B in the OpenHands scaffold. We log signatures at every step. We then estimate, from the logged grid, the marginal number of problems resolved per thousand tokens under two options: one more round on the current attempt, or one new attempt. The question is whether $c$ and signature stall separate the regions where each option wins. Because SWE-Gym Lite is disjoint from the evaluation benchmarks, this also calibrates the rule's two thresholds without touching test data.

Signatures must come only from what the agent can legitimately run. On SWE-bench these are the repository's existing tests and reproduction scripts the agent writes itself, never the hidden evaluation tests. On LiveCodeBench they are the public examples.

The rule itself is a few lines of control logic. Deepen while the attempt's signature keeps changing and does not regress. Widen when the signature has stayed the same for two rounds and $c$ exceeds its threshold. When widening, draw the new member so its expected failures differ. Concretely, prefer a pool model whose past signatures overlap least with the currently dominant failure; failing that, use a fresh context with the dominant traceback withheld. Stop when at least two passing candidates produce identical outputs on agent-generated probe inputs. Otherwise, choose among passing candidates with the SWE-Gym verifier.

Baselines

Naive reference. A single attempt with one pass and no feedback (Pass@1) anchors the curve. SWE-Gym reports 20.6% on SWE-bench Verified for its fine-tuned 32B agent.

Fixed-budget deepening. Single-agent self-debugging for 3 and 6 rounds with MEIC-style rollback. This is the cheapest serious competitor. Its known plateau after 2–3 rounds is exactly what the rule claims to escape.

Strongest open-weights selection baseline. SWE-Gym Best@16 with its 32B verifier, which reports 32.0% resolve rate on SWE-bench Verified (SWE-Gym). We rerun it with the same backbone and scaffold. We report Best@k at every $k$, so our method is compared at equal tokens as well as at $k=16$.

Strongest structured committee. Adversarial Review, which reports 75.2% on SWE-bench Verified and 87% on LiveCodeBench against 72.6% and 82% for five-agent MARS (Adversarial Review). Those figures come from its own, stronger backbone. We will not compare against them directly. Instead we reimplement the protocol on our backbones and compare at matched tokens. We also report the result of our rule applied to the Adversarial Review committee itself, where the rule decides when the Reviewer and Critic are invoked at all.

Trained-policy reference. BOOSTAPR reaches 40.7% on SWE-bench Verified with Qwen-2.5-Coder-32B after reinforcement learning (BOOSTAPR). This number is for context only, since our rule is training-free and can sit on top of such a policy. We test that as well if its weights are available.

Fairness. Every method gets the same backbone pool, scaffold, per-problem token cap, and tuning budget. All thresholds for all methods are tuned only on SWE-Gym Lite.

What counts as beating a baseline. Our method beats a baseline if it achieves an equal or higher resolve rate at no more than half the tokens, or a higher resolve rate at equal tokens, with 95% bootstrap intervals over problems.

The decisive ablation. We replace the signature and $c$ with a bare pass/fail trigger: widen whenever the current attempt still fails after two rounds. This isolates whether the content of execution failures matters or merely their presence.

Why not just…?

Why not just trigger extra compute whenever tests still fail? This is the most serious threat, and it is essentially the "Adaptive AR" the Adversarial Review authors suggest. If the pass/fail-trigger ablation matches the full rule on the accuracy–token curve, the claim about correlated failures is wrong. We would then report adaptive triggering as sufficient. The claim survives only if widening conditioned on high $c$ recovers problems the bare trigger wastes tokens on. Concretely, we will count problems where the bare trigger widens with near-duplicate attempts that fail the same way.

Why not just sample 16 attempts and filter by tests or a verifier? Fixed breadth spends 16 trajectories on problems a single attempt solves. It also leaves a 10.8-point gap between Pass@16 and Best@16 (SWE-Gym). Our rule keeps that same verifier for the final pick, so any gain comes from allocation, not selection. The per-$k$ curve will show whether allocation matters beyond simply choosing a smaller fixed $k$.

Why not just keep refining one attempt longer? The plateaus reported by CodeTransOcean and ICER, and the 29.4% stuck-in-loop rate in SWE-Gym, predict that pure depth saturates. SlopCodeBench predicts it also degrades code quality. The six-round deepening baseline tests this directly, on both accuracy and erosion.

Why not just use a bigger model? MEIC shows workflow lifting GPT-3.5 to roughly GPT-4 syntax-fix rates. Still, scale may shrink the room for allocation. We therefore run the rule on the smaller backbone and on a larger one, and report whether its token savings persist.

Proposed Experiments

Main comparison. The main result is accuracy against tokens on SWE-bench Verified (500 issues) and SWE-bench Lite. Transfer is checked on LiveCodeBench, where we have no measured grid, and on Defects4J v2.0, a Java benchmark, to test whether signatures work across languages.

Mechanism check. Across all logged steps, we test whether high $c$ with a stalled signature predicts that widening beats deepening. We report the area under the curve of that prediction against the grid's ground truth.

Secondary claim. On SlopCodeBench's 196 checkpoints, we compare the erosion and verbosity measures of the final code under the rule against six-round deepening at equal tokens. The prediction is that erosion grows less often than in the 77% of trajectories the benchmark reports, because widening restarts from clean code.

Falsification conditions. The proposal fails if the pass/fail ablation matches the full rule, or if $c$ carries no predictive signal on SWE-Gym Lite. Either outcome is informative. It would mean the plateau in refinement is not caused by correlated errors, and would redirect effort toward better verifiers.