Debate That Must Fail a Test: Execution-Admitted Objections for Adaptive Code Review

When a language model writes or repairs code, a second model arguing with it can find root causes that the first model misses. The same argument also burns tokens on every task and invents problems that are not there. This proposal makes one rule change to multi-agent code debate. An objection from the critic counts only if the critic can turn it into a small executable check, such as a test, an assertion or a reproduction script, and that check actually fails on the current patch. Debate stops as soon as a round produces no new failing check. Easy tasks then exit after one cheap round because the critic cannot break them. Hard tasks keep debating for as long as the argument keeps exposing real failures. Speculative concerns that cannot be reproduced never reach the code.

The opportunity is concrete. Adversarial Review (AR) runs a three-agent protocol: a main agent edits the code, while a reviewer and a critic debate in text with the code frozen. It reports 75.2% on SWE-bench Verified against 71.6% zero-shot, and 87% against 77% on LiveCodeBench. It pays roughly 4.5 times the tokens of a single agent because every change triggers the full debate. It also suffers "scope creep": persistent debate on hard issues produces over-engineered patches that break hidden tests. The AR authors name "Adaptive AR" as future work and do not build it. The obvious version of that future work is "add a confidence gate before AR." We argue it is the wrong target. A gate placed before the debate cannot fix scope creep, which happens inside the debate. Execution evidence placed inside the debate addresses cost and scope creep with the same mechanism.

Core Claim

Requiring each debate objection to be backed by a failing executable probe, and stopping when no new failing probe appears, gives two things. Pass rates should match or exceed uniform adversarial review at a fraction of its token cost. Fewer patches should regress hidden tests. The amount of debate a task receives is then set by how breakable the patch is, measured in a sandbox, rather than by the model's own confidence.

This differs from the closest work in where the decision to keep spending comes from. LASE decides from a leader agent's token log-probabilities and entropy. Debate Only When Necessary (DOWN) decides from softmax or verbalized confidence. Both were tested only on text and math, and both decide once, before debate, with a fixed debate shape once triggered. AR debates in text only. SWE-Debate runs three rounds unconditionally on every issue. On the compute side, ADAP allocates verification by reward-model rank and never reads what failed. Dynamic Scaling of Unit Tests (CodeRM) scales the number of generated tests by estimated difficulty for best-of-N selection, with no adversary and no iterative editing. Our proposal takes test generation from CodeRM and the citation-constrained critic from AR. It differs from both because the tests are written adversarially, in response to a specific patch, and their pass/fail outcomes drive both admission and stopping.

Mechanism

The main agent produces a patch and runs whatever tests are available: visible unit tests for competitive programming, or the repository's existing tests plus an issue reproducer for SWE-bench. The critic then raises objections in AR's categorical format. Any objection that would change code must come with a probe. A probe is a short test or script that, the critic claims, fails on the current patch.

Each probe is run in the sandbox. Only probes that actually fail are admitted to the outer loop. A second filter targets scope creep: a probe is admitted only if, per coverage tracing, it exercises lines the patch touched or functions on the issue's call path.

Before the main agent edits, the reviewer may contest an admitted probe as encoding the wrong specification. It must cite the issue text or docstring, and contested probes are dropped. The main agent then edits to fix the admitted probes. Following the rollback idea in Fang et al., an edit is reverted if it breaks any previously passing test.

The stopping rule is simple: the loop terminates on the first round with zero newly admitted probes, or at a hard cap. Spending is therefore driven by evidence at every round, rather than set by a gate before the first round.

Two lessons from efficient video recognition shape the design. FrameExit shows that exit decisions work best when tied to an observable signal of whether more computation changes the answer. Here that signal is "did a new check fail." LiteEval shows that skipping expensive paths without keeping state synchronized collapses accuracy (80.0% to 65.7% mAP on FCVID). So admitted probes persist as a growing regression suite carried across rounds, rather than as prose kept in the context window. This also limits the context bloat that iterative refinement suffers.

The same video literature warns that a gate's own cost can eat its savings. Probe writing and execution are this method's gate cost, and we report them separately.

Research Program

We will first reimplement AR faithfully with a fixed open-weight backbone and confirm we are close to its reported gains over zero-shot. The file does not give AR's backbone, so absolute numbers may differ. The comparisons that matter are relative ones under one backbone.

We then build the execution-admitted variant on the same agents and prompts, changing only admission and stopping. The analysis has three parts. The first is the per-task distribution of debate rounds and tokens, and how well it tracks difficulty proxies: LiveCodeBench difficulty labels, and patch size or number of files in the gold patch for SWE-bench. The second is the scope-creep rate, defined as the fraction of final patches that pass the reproducer but fail hidden tests that the pre-debate patch passed. The third is the precision of admitted probes: how often a failing probe reflects a genuine bug, judged against hidden tests and the gold patch.

Baselines

The naive reference point is the single main agent run zero-shot. Above it sits single-agent self-repair that feeds sandbox errors back for up to three rounds. This is the standard closed-loop baseline; LiveCodeBench reports self-repair lifting GPT-4-Turbo from 24.5% to 36.9%. Both run on the same backbone and sandbox.

The strongest existing method on this question is uniform Adversarial Review. It reports 75.2% on SWE-bench Verified and 87% on LiveCodeBench, beating zero-shot (71.6%, 77%) and the five-agent MARS team (72.6%, 82%), at about 4.5 times single-agent tokens (Adversarial Review). We run it with identical agents, round cap and prompt-tuning budget. Beating it means matching its pass rate within noise at no more than half its tokens, or exceeding it at equal tokens. For SWE-bench Verified we also report SWE-Debate as a fixed-round debate reference; it reports 41.4% Pass@1 with DeepSeek-V3. We rerun it on our backbone where feasible.

Two adaptive competitors are ported from text reasoning to code. The first is a DOWN-style gate: debate is triggered only below a confidence threshold, with the threshold tuned on a held-out split. DOWN reports cutting agent calls from 6.0 to 1.02–1.96 on its text tasks. The second is a LASE-style entropy gate using white-box log-probabilities. LASE reports 0.4k against 5.6k tokens on GSM-Symbolic. For both we give the threshold the same tuning budget as our relevance filter.

A compute-matched CodeRM-style baseline spends our method's measured token budget on generating more unit tests and selecting among self-repair candidates, with no critic.

The ablations isolate the key ingredient. One keeps our stopping rule but admits text-only objections. One admits failing probes without the coverage relevance filter. One runs our admission rule under AR's fixed round count. If the text-only-admission ablation matches the full method, execution grounding is not doing the work.

Why not just…?

Why not just skip debate whenever the visible tests pass? This is the most serious threat. On LiveCodeBench, visible tests may already separate easy from hard tasks well enough that a pass/fail gate in front of AR recovers most of the savings. It does nothing about scope creep, though, which arises on tasks that fail initially and then get debated into over-engineering. It also misses tasks that pass visible tests and fail hidden ones, which is exactly where AR found deeper bugs. We include this gate as a baseline and report results by initial-pass status. If it matches our method on both cost and hidden-test pass rate, the idea reduces to that gate, and we will report that outcome.

Why not just port DOWN's confidence gate to code? Confidence gates fail in the miscalibrated-overconfidence regime: a model confidently wrong about an edge case never triggers debate. They also decide only once, so they cannot end a debate that has drifted into speculation. The DOWN and LASE ports show whether confidence predicts which tasks benefit from debate. We plot pass-rate gain from debate against both confidence and our probe-failure signal.

Why not just spend the saved tokens on more tests and best-of-N? CodeRM's gains (53.58% to 72.01% on HumanEval Plus with a Llama3-8B policy) show non-adversarial test scaling is strong. Its dynamic allocation added only 0.5% over equal allocation on MBPP Plus, which suggests difficulty-scaled test counts alone are not where the gains come from. The compute-matched baseline settles whether adversarial, patch-specific probes find bugs that generic tests miss.

Why not just cap AR at one round? A cap cuts cost uniformly and does not discriminate between tasks, so it should hurt the hard tasks where AR's deeper root-cause fixes come from. The fixed-round ablation tests this directly.

Proposed Experiments

The main benchmarks are SWE-bench Verified and LiveCodeBench, chosen because AR reports numbers on both. We add SWE-PRBench for review quality, where AR reports an F1 of 0.533. On every benchmark we report pass rate, total tokens and wall-clock time, with the gate cost (probe writing and execution) broken out. We also report scope-creep regression rate and patch churn in changed lines. Results are summarized as cost–accuracy Pareto curves across round caps.

The method is supported if four things hold. It sits on or above AR's curve at well under 4.5 times single-agent tokens. Its regression rate is lower than AR's. Rounds per task correlate with difficulty proxies. The text-only-admission ablation clearly underperforms.

The method is falsified under two conditions. One is that the visible-test gate or the DOWN port reaches the same Pareto curve. The other is that admitted probes are often wrong, so that encoding wrong expectations drives regressions, which would show as low probe precision.