When language models fix code in a loop, a second model is often asked to review each attempt. That reviewer is persuasive but unaccountable. It can flag bugs that do not exist, approve patches that are broken, and push the coder into editing parts of the program that were working. Human peer review at large venues handles a version of the same problem by adding a middle layer. An area chair, a person who reads all the reviews of one paper, turns them into a single, bounded list of required changes. A "shepherd" then checks that each required change was made before the paper is accepted. We propose to build that middle layer for code repair agents, with one change that only software allows. The meta-reviewer may put a reviewer's concern into the list of required changes only if the concern comes with an executable witness: a small test or reproduction script that fails on the current patch. The shepherd accepts the revision only when every witness now passes, the previously passing tests still pass, and no code outside the mandated region has been touched.
In two sentences for a colleague: current multi-agent code review lets agents argue in natural language and then lets the coder edit freely, which produces false agreement and over-editing. We add an area-chair agent that keeps only those review comments backed by a test that fails, and a shepherd that confines and verifies the revision against exactly those tests. The bet is that most of the value of "hierarchy" in peer review comes from the chair converting noisy critique into a bounded, checkable contract, and that in code this contract can be made mechanical.
Core Claim
The closest existing system is Adversarial Review (AR). It freezes the code while a Reviewer and a Critic argue under a structured-disagreement format that cites line numbers, and only then lets the Main Agent edit. AR reaches 75.2% on SWE-bench Verified against 71.6% zero-shot and 72.6% for the five-agent MARS system. Its authors report two failures that motivate this proposal. The first is over-editing: the coder touches unbroken code and breaks tests that used to pass. The second is a token cost of roughly 4.5 times zero-shot. AR's Reviewer–Critic consensus is still verbal. The Critic can demand evidence, but "evidence" means cited lines, not a failing execution.
Articulate Intuition or Genuine Analysis? builds a benchmark of LLM judges on scientific peer review. It shows that such judges reward fluency, structure and length over actual fault-finding; length explains about 74% of the gap between agentic and human review scores. The authors name the lack of execution access as their main limitation. Code review is the setting where that limitation can be removed.
Our claim is that making execution witnesses the admission criterion for review comments, with a shepherd that enforces scope, will do three things. It will reduce regressions on previously passing tests relative to AR. It will match or exceed AR's resolve rate. It will do so at lower token cost, because unwitnessed comments are discarded before they trigger edits.
We deliberately do not claim that a deep chain of agents in the roles of program chair, senior area chair, area chair and reviewer helps in itself. The file's mapping from publishing hierarchies to agents suggests such a tree. Nothing in the evidence shows that depth, rather than the chair's filtering function, is what matters, so depth is treated as an extension to test.
Relation to Prior Work
Three systems supply pieces of the method.
- AR supplies the frozen-code inner loop and the adversarial Critic, which we keep.
- MAGIS splits work among a Manager, Developers and a QA Engineer. It reaches 25.33% on SWE-bench Lite against 18.00% for SWE-Agent. Its manager decomposes the task before coding but reviews flatly afterward. We borrow its file-level assignment for multi-file issues.
- MEIC pairs a debug agent with a scorer and rolls back to the last stable state on simulator regressions. It fixes 78% of functional RTL errors with GPT-4. Its rollback decision is binary and whole-patch. Ours is per-hunk and tied to specific witnesses, which is closer to the graded outcomes (conditional accept with shepherd, major revision, reject and resubmit) that the PoPETs editors' introduction describes.
Other work motivates the design or constrains it.
- Returns from repair loops flatten quickly. In CodeTransOcean, ChatGPT rises only from 48.57% to 52.57% over three self-debugging rounds. More rounds are not the answer; better-targeted rounds might be.
- Confident loops can make code less safe. Do Users Write More Insecure Code with AI Assistants? finds that 36% of AI-assisted participants wrote SQL-injectable code against 7% of controls, with inflated user trust. Reviewer confidence therefore should not stand in for evidence. The same result also warns that functional tests alone miss security flaws, a limitation of witnesses that we return to below.
- Human structures add a middle tier to bound each person's workload. The TNSE editorial splits the board into technical pillars, and the OJCS editorial introduces senior editors. These are organizational reports, not controlled experiments, and we treat them as inspiration only.
Research Program
The system has four roles over a shared sandbox.
- Coder. Produces a patch.
- Reviewers. One or more reviewers comment on the patch while it is frozen. AR's Critic challenges each comment.
- Meta-reviewer (the area chair). Asks, for each surviving comment, that a witness be written. A witness is a test or script that runs in the sandbox and fails on the current patch because of the stated concern. The meta-reviewer then emits a mandate: the set of witnessed concerns, and the file-and-hunk region the coder may edit to address them. Comments whose witness cannot be made to fail are dropped. They are logged as "unwitnessed" so we can later measure whether they were real.
- Shepherd. After the coder revises, the shepherd runs:
- all witnesses,
- the repository tests that passed before the patch,
- a diff-scope check against the mandate.
A revision that fixes some witnesses but breaks a previously passing test, or strays outside scope, is not rolled back wholesale. The offending hunks are reverted and the remaining hunks are kept, which is the per-hunk rollback.
The loop ends when there are no witnessed concerns left, or when a fixed round budget runs out.
Guarding the witnesses. Witnesses are agent-written, so their quality must be controlled. A witness that fails for unrelated reasons, such as import errors or wrong assumptions about the API, is a false witness. The meta-reviewer must therefore also show the witness passes on the pre-patch repository when the concern is about a regression, or give a stated reason why no such check applies. SWE-bench's hidden evaluation tests are never shown to any agent.
The analysis this enables. Every comment ends up labeled as witnessed or unwitnessed, so the pipeline yields a dataset for studying the "form versus function" question in code review. We can score comments with the Kahneman4Review rubric and ask whether rubric scores predict witnessability, and whether witnessability predicts real bugs.
Extension for large changes. For issues that change many files, we test a pillar variant. Separate meta-reviewers each own a group of files, and one top-level shepherd checks the integrated patch. This is the only place where the "hierarchical" part of peer review is tested beyond the single area-chair tier.
Baselines
Zero-shot. A single generation of the same model with no review loop, run through the same harness. This is the do-nothing reference point. AR reports 71.6% on SWE-bench Verified and 77% on LiveCodeBench for its zero-shot setting. We will rerun it with our model rather than rely on those figures.
Self-repair on execution feedback. One model receives its own test tracebacks and retries up to the same token budget as the full system. This is the standard loop studied in LiveCodeBench, where GPT-4-Turbo self-repair improves from 24.5% to 36.9% with interpreter feedback. It tests whether a reviewer is needed at all.
AR. The strongest existing method, reimplemented with the same model, sandbox and number of rounds. On SWE-bench Verified it reports 75.2% pass rate, on LiveCodeBench 87%, and on SWE-PRBench a review F1 of 0.533 AR. It is the right competitor because it already solves false consensus with an adversarial critic, so any gain we show must come from witnessing and shepherding rather than from having a critic. Fairness rules:
- Both systems get the same prompt-tuning budget, measured in development-set runs.
- Both are compared at matched average tokens per issue as well as at their natural budgets.
- Both are run with three seeds.
Beating AR means an equal or higher resolve rate, with a statistically significant reduction in broken previously-passing tests, or the same resolve rate at clearly lower tokens.
AR plus scope lock and test rollback. AR with a mechanical diff restriction to the files of the original patch and whole-patch rollback when prior tests break. This is our own ablation and the most dangerous baseline, discussed below. It isolates whether gains come from witnesses or merely from scope enforcement.
Unwitnessed meta-review. Our full pipeline with the witness requirement removed. The meta-reviewer still synthesizes comments and the shepherd still checks scope and prior tests, but concerns enter the mandate on the meta-reviewer's judgment alone. This ablation isolates the key ingredient.
MAGIS. Reported at 25.33% on SWE-bench Lite. It is included on Lite as a reference for the multi-file pillar variant, rerun with our model where feasible.
Why not just…?
Why not just give one strong model the test tracebacks and let it self-repair? Existing tests only catch what they already test. The failures we target are over-editing and plausible-but-wrong fixes that local tests do not expose. The flat gains in CodeTransOcean suggest that extra self-repair rounds add little once the visible errors are gone. The matched-token self-repair baseline settles this directly. If it matches our system on resolve rate and regressions, the review machinery is not earning its cost and we will say so.
Why not just add a scope lock and rollback to AR? This is the real threat. Much of AR's reported over-editing might disappear with a mechanical rule that the coder may touch only the originally patched files, plus a revert whenever prior tests break. For that reason the shepherd's scope and regression checks are shared with this baseline, not claimed as ours. What the scope lock cannot do is keep a hallucinated comment from sending the coder off to change correct code inside the allowed files, and it cannot surface concerns the existing tests miss. If "AR plus scope lock" equals the full system, the witness idea has failed on resolve rate. We would then report whether it still wins on tokens, since dropping unwitnessed comments should cut edit rounds.
Why not just build the full program-chair, senior-area-chair, area-chair tree, since that is the actual peer-review structure? Nothing in the file separates the benefit of depth from the benefit of synthesis. AR's own result, where five flat agents do worse than three, warns that more agents add overhead. We therefore make one synthesis tier the core claim and test pillars only on the multi-file subset, where MAGIS reports that flat fan-out breaks down.
Why not just use a stronger reviewer model? The Kahneman4Review findings suggest that stronger LLM judges still reward form. We will run the unwitnessed ablation with a larger reviewer model and check whether its false-comment rate drops to the witnessed level.
Proposed Experiments
Primary evaluation. SWE-bench Verified, the 500-issue subset of SWE-bench, using three metrics:
- resolve rate;
- the rate at which previously passing tests (SWE-bench's PASS_TO_PASS tests) are broken;
- lines edited outside the files touched by the reference patch, as a scope-creep measure.
Code generation. LiveCodeBench, whose problems are released after the model's training cutoff, to confirm that gains are not specific to repositories.
Review quality. SWE-PRBench, comparing the F1 of witnessed comments against AR's 0.533. The hypothesis is that witnessing raises precision, possibly at some cost to recall.
Multi-file extension. SWE-bench Lite instances whose reference patch touches more than one file, where the pillar variant is compared against single-meta-reviewer and MAGIS-style runs.
Hardware transfer. The MEIC RTL debugging set, with ModelSim simulation as the execution environment. Its 78% GPT-4 functional fix rate is the reference, and the question is whether per-hunk rollback beats whole-patch rollback there.
Decisive check on witnesses. The central test is whether witnessing tracks real bugs. Against the hidden SWE-bench tests and reference patches, which agents never see, we measure how often witnessed and unwitnessed comments point to real defects. A comment points to a real defect if the reference patch changes the flagged behavior, or if the hidden tests fail on the flagged hunk. If unwitnessed comments are as predictive as witnessed ones, the filter is discarding signal and the core claim is false. We also report the rate of false witnesses, meaning witnesses that fail for unrelated reasons, since a high rate would make the filter noise.
Decisive check on tokens. We plot resolve rate against tokens for every system. The proposal succeeds only if the witnessed system sits on or beyond AR's reported efficiency frontier. That means the ablation "AR plus scope lock" must lose on at least one of three axes: resolve rate, regression rate or cost.
Known gap. Security concerns, such as the injection flaws documented by Perry et al., are often hard to witness with unit tests. We will report separately how many security-flavored comments go unwitnessed, as a named limitation of the approach.
References
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Editors' Introduction
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Do Users Write More Insecure Code with AI Assistants?
- Message From the Editor-in-Chief
- Editorial
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- SWE-bench: Can Language Models Resolve Real-world Github Issues?