Coding agents now routinely produce repository-level patches that pass their functional tests and still contain exploitable flaws. On a benchmark mined from real security fixes, one strong agent passed functional tests 57% of the time, yet only 11.8% of its solutions were secure (Is Vibe Coding Safe?). Asking several language models to review each other's code is the obvious fix, but such panels tend to agree in order to finish rather than to be right, and longer debates often make answers worse. Human expert panels faced the same failure decades ago. The Delphi method was their answer: experts rate items anonymously and independently, see only the group's summary statistics between rounds, and anyone far from the group must either move toward it or write a justification.
We propose a Delphi panel of reviewer agents with one change that a sandbox makes possible. A dissenting reviewer may not justify its position with prose. It has to submit a test that a coordinator then runs inside an isolated container. A reviewer who claims a vulnerability must supply an attack test that fails on the patch. A reviewer who claims the patch is safe against a flagged attack must supply a test showing that the attack does not succeed. The panel sees only anonymized rating distributions and the execution results of these tests. The patch may be modified only in response to an attack test that reproduces. The rule that gives a modified patch its final yes or no is a classical Delphi agreement statistic, and its threshold is calibrated against sandbox ground truth rather than chosen by convention.
Core Claim
Making executed tests the only accepted form of dissent should fix two failures that existing multi-agent code review leaves open. The first is false consensus: reviewers abandon correct objections because a peer disagreed. The second is scope creep: speculative objections push the author into unrequested changes that break hidden tests. The specific prediction is that on repository-level security tasks this protocol raises the fraction of solutions that are both functional and secure, at the same token and sandbox budget as the strongest alternatives, without the drop in functional correctness that security prompting causes. A second, smaller claim runs in the other direction. The sandbox resolves a long-standing weakness of Delphi studies, namely that agreement cutoffs such as the content validity ratio are set by convention (VISION-Cog and trauma follow-up services; Follow-up after major traumatic injury: a survey of services in Australian and New Zealand public hospitals apply such indices with fixed cutoffs). Here the cutoff can be fitted to observed exploit outcomes.
Relation to Prior Work
Four papers supply partial mechanisms, and each leaves out the piece we add. Voting or Consensus? compares seven decision protocols on text reasoning benchmarks, without code or execution. It introduces blind first drafts, which add 3.3% accuracy on average, and it documents "drift," in which more rounds lower accuracy. It also finds that consensus beats voting on knowledge tasks but loses badly on reasoning, 28.4% versus 54.8% on MuSR. We read this as a constraint: consensus over text cannot be trusted on hard reasoning. That is why our protocol grounds every change of position in an executed test, and why it stops on rating stability rather than on a fixed round count.
Adversarial Review runs a reviewer and a critic in direct dialogue. The critic must tag each objection as agreement, evidence-backed disagreement, or speculative concern. The paper names false consensus as the core pathology and reports scope creep as a remaining failure. We borrow its insistence on evidence. We differ in three ways: the evidence must be executable rather than a quoted code span, reviewers never see each other's identities or argue directly, and the decision comes from a statistic rather than from the conversation ending.
SWE-Debate uses independent ranking followed by open cross-examination, then a single discriminator writes the final plan. It targets functional bug fixing only. Multi-LLMSecCodeEval shows that collaboration among language models alone helps little, and that adding a static analyzer to an ensemble helps a lot. That finding motivates grounding the panel in tools. It also means our protocol has to beat a tool-filtered ensemble, not only a talking one.
On the sandbox side, OpenHands supplies the Docker runtime in which tests are executed. Catching Infrastructure Sabotage shows a structural monitor catching covert sabotage. We use its setting as a transfer test.
Research Program
The protocol runs after an author agent produces a candidate patch in an OpenHands container. A panel of $N=5$ reviewer agents, drawn from at least two model families, independently inspects the patch. Each reviewer marks regions it considers risky and rates each region on a 1–9 scale for "contains an exploitable vulnerability." The coordinator merges overlapping regions and returns to each reviewer only the median and interquartile range of the ratings per region, never the identities or the rationales behind them.
A reviewer counts as an outlier on region $k$ if its rating lies more than one interquartile range from the median. An outlier must do one of two things. It can re-rate toward the group. Alternatively, it can submit a witness test, meaning an attack test if it rated high or a defense test if it rated low.
The coordinator runs each witness in a fresh container. A witness is valid only if it meets three conditions:
- it calls only interfaces exposed in the task description;
- a blind majority of the other reviewers judges that its assertion expresses a security property rather than a functional preference;
- it is deterministic across three reruns.
Validity is checked before the coordinator interprets the result. An attack test reproduces when it is valid and fails on the patch. Only reproduced attack tests are forwarded to the author, which may then edit the patch. Every edit must keep all visible functional tests green. This is the scope-control rule: the author cannot act on a speculative objection.
The next round shows reviewers the updated distribution and the pass or fail outcome of every witness, again anonymized. Rounds stop when, for two consecutive rounds, every region's median is unchanged and its interquartile range shrinks by less than one scale point, or when a cap of $T=4$ rounds is reached. The stability rule is borrowed from Delphi practice and is meant to prevent the drift that Voting or Consensus? documents.
For the final verdict, each reviewer casts a binary judgment of "secure" on the final patch. We compute the content validity ratio
$$\mathrm{CVR} = \frac{n_s - N/2}{N/2},$$
where $n_s$ is the number of reviewers judging the patch secure. The patch is accepted if $\mathrm{CVR} \ge \tau$ and no valid attack test reproduces against it. The threshold $\tau$ is fitted on a development split to maximize the secure-and-functional rate.
The witness requirement lets the sandbox resolve every dispute that can be resolved by execution. The statistic handles the remaining risk that no reviewer could turn into a test.
Two measurements come out of the program besides headline accuracy. The first is a false-consensus rate: the fraction of accepted patches that the hidden security tests later show to be vulnerable, in a region some reviewer flagged in round one and later abandoned without a witness. The second is a calibration curve. It asks whether the panel's round-one interquartile range predicts hidden vulnerabilities, which would make panel disagreement a cheap triage signal even without the full protocol.
Baselines
The naive reference point is a single agent with no review. We use SWE-agent and an OpenHands CodeAct agent run on SUSVIBES with the same backbone models as our panel. The cited figure for this regime is 57% functional and 11.8% secure for SWE-agent with Claude 4 Sonnet (Is Vibe Coding Safe?). We rerun it rather than copy it, because our backbones may differ.
The strongest existing tool-grounded method is the hybrid pipeline from Multi-LLMSecCodeEval. It generates with an ensemble, filters with CodeQL, and cross-verifies with a second model. It is the strongest competitor because it already combines several models with a deterministic tool. It reports 97.32% on SecLLMEval and 99.06% on SecLLMHolmes. No reported result found on Lacuna for this pipeline on SUSVIBES. We will port it to SUSVIBES with the same models and the same token budget, and also allow it the same number of candidate generations as our panel has reviewers.
The strongest existing deliberation method is Adversarial Review, re-pointed at security review. It reports 75.2% on SWE-bench Verified and 87% on LiveCodeBench. No reported result found on Lacuna for it on SUSVIBES or on any security pass rate. We run it with the same backbones and the same container, matched on total tokens, and with sandbox access granted so that it is not handicapped on execution.
We also include an "All-Agents Drafting plus vote" baseline from Voting or Consensus?: blind independent reviews followed by one majority vote, with no iteration.
The ablations isolate the key ingredients:
- Text Delphi: outliers justify in prose, exactly as in classical Delphi.
- Open Delphi: identities and full rationales are shown, so anonymity is removed.
- All witnesses: every reviewer must produce witnesses every round, so that compute is no longer routed to outliers.
- Fixed rounds: four rounds, with the stability stopping rule removed.
Beating a baseline means two things on the held-out SUSVIBES split. The secure-and-functional rate must rise, with the 95% bootstrap interval over tasks and three seeds excluding zero. The functional pass rate must not fall by more than two points. Total tokens plus sandbox seconds must stay within 10% of the baseline's.
Why not just…?
Why not just have one agent write its own attack tests in the sandbox? This is the most serious threat, because it keeps the executable evidence and drops the panel. It is plausible that most of our gain would come from execution alone. We test it directly with a single-agent self-red-teaming baseline. One agent generates attack tests against its own patch until it hits the same token and sandbox budget as the panel, then fixes only reproduced failures. If that matches our protocol, the Delphi structure is not earning its keep, and we will say so. Our prediction that it falls short rests on Is Vibe Coding Safe?: stronger reasoning in a single model did not bring better security, which suggests that one model's blind spots also limit which attacks it thinks to write. Independent reviewers from different model families should flag a wider set of regions.
Why not just run an ensemble with a static analyzer, as the hybrid pipeline does? It nearly saturates SecLLMEval, so on snippet-level prompts it may well be enough. Purple Llama CyberSecEval and Multi-LLMSecCodeEval both note that analyzers miss logic-level and cross-file weaknesses. SUSVIBES spans 79 weakness types across multi-file tasks, which is where we expect the gap to appear. We also run our protocol with the analyzer's findings shown to reviewers as an extra input, so the comparison does not pit execution against analysis.
Why not just prompt the agent to prioritize security? The SUSVIBES study found that vulnerability hints slightly improved safety but caused a significant drop in functional correctness. We include this nudged agent as a baseline mainly to show that our edit-only-on-reproduced-attack rule avoids that penalty. If our functional rate also drops, the scope-control claim fails.
Why not just sample many patches and pick the one passing the most tests? Independent samples from models trained on the same public code tend to share the same memorized insecure pattern. Selection by functional tests is therefore blind to the flaw. We run best-of-$N$ with $N$ matched to the panel's compute. If it closes most of the gap, the panel's value lies only in which tests get written, and the self-red-teaming comparison above would then decide between the two.
Proposed Experiments
The main experiment uses SUSVIBES, with its 186 tasks split into 46 for development, used only to fit $\tau$ and the outlier rule, and 140 held out. Metrics are the secure-and-functional rate, the functional and security rates separately, the false-consensus rate defined above, tokens, and container seconds.
For breadth, we run CyberSecEval 2 with its security pass rate, comparing against the 9.55–14.92% C/C++ gain that Stream of Revision reports over plain decoding. We expect little advantage there, because fixes in that benchmark are local.
As a transfer test, we use ControlArena iac_fast from Catching Infrastructure Sabotage. Here the panel acts as a monitor, and its witnesses are policy-checking tests run against the deployed infrastructure template. The reference point is that paper's asynchronous monitor, with 3.5% false negatives at 1% false positives, together with its honest-task success of about 82–83%, which a useful monitor must preserve.
Three checks decide the outcome:
- Witnesses must matter. If Text Delphi matches the full protocol, the claim about executable dissent is wrong.
- Self-red-teaming must lose. If the single agent equals the panel at matched budget, the Delphi structure adds nothing beyond execution.
- The fitted threshold must transfer. A $\tau$ calibrated on the SUSVIBES development split must stay near-optimal on the held-out split. If it does not, the sandbox has not actually resolved Delphi's threshold problem.
References
- Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
- Feasibility, comprehensibility and acceptability of the VISION-Cog, a novel tool to assess cognitive impairment in visually impaired older adults: a cross-sectional pilot study in Singapore
- Follow-up after major traumatic injury: a survey of services in Australian and New Zealand public hospitals
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution
- Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- Catching Infrastructure Sabotage When Coding Agents Are Insider Threats
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Models
- Autoregressive, Yet Revisable: In Decoding Revision for Secure Code Generation