Automated peer review is starting to be run as a small editorial board. Several reviewer models each write a report, and a "meta-reviewer" model plays the area chair, reading the reports and issuing a decision. These systems already have a documented blind spot. When a paper's soundness is damaged, for example when the experiments that back a practical claim are removed, the meta-reviewer lowers the paper's score only mildly and usually blames the presentation. The same blind spot matters well beyond automated review. A large body of theory work announces practical speedups or real-world gains that no experiment supports, and editors have no routine way to flag the mismatch between what a paper claims and what it shows.
This note argues that the failure lies more in how the board is wired than in what any single model can see. Even when one reviewer notices that a claim has no supporting evidence, that observation is averaged in with fluent, positive reports and diluted before it reaches the decision. We propose to measure this "detection-versus-propagation" gap directly. We then test a single structural change: a separate evidence-audit tier that reads only claims and the evidence offered for them, and whose finding can hold up a decision rather than being averaged into it.
Core Claim
In tiered automated review, the share of papers whose missing empirical support is noticed by at least one agent is much larger than the share whose final decision reflects that flaw. Closing this gap depends mainly on two choices: which inputs the auditing agent sees, and whether its verdict is treated as a vote or as a gate. Better prompting matters less. The test that separates this claim from "just prompt better" is an ablation. We hold the auditor's text fixed and change only how the board consumes it. If routing alone moves flaw propagation substantially while the auditor's output stays constant, the structural claim holds. If it does not, the claim is false.
Grounding in Prior Work
The motivating evidence comes from Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review. That study applied 79,302 targeted edits to 508 ICLR 2024 papers, their reviews, and their rebuttals, and ran GPT-4o and Gemini 2.0 Flash as reviewers and meta-reviewers. It reports three things relevant here. Soundness degradation is penalized mildly and misattributed to presentation, which the authors call the "Presentation Trap." Hype is rewarded. Flipping a single review's conclusion to "Strong Reject" lowers the meta-review score by an average of 1.65 points on a 10-point scale. Read together, these findings say the meta-reviewer responds strongly to a reviewer's verdict but weakly to the substance underneath it. That pattern is exactly what an aggregation failure would produce. The study proposes no defense, and our proposal begins at that point.
AgentReview simulates the full five-phase conference hierarchy over 523 ICLR papers. It shows that area-chair style alone changes 30.6% of decisions, that discussion compresses rating variance by 27.2%, and that its agents cannot verify empirical claims. We reuse its hierarchy as a baseline. Its result that board structure moves outcomes motivates our structural focus, but AgentReview never asks which structure preserves a specific, known flaw.
PeerArg decomposes reviews into aspect-level arguments and aggregates them with formal argumentation semantics. It reaches Macro F1 of 76.64% on the PRA dataset against 72.5% for an end-to-end Mistral-7B baseline. We borrow its aspect decomposition as the strongest "structured aggregation" competitor. Because PeerArg still aggregates by weighted support and attack, a lone soundness objection can be outweighed, which is the behavior we predict fails. Multi-agent peer review collaboration shows that a flat, mutual-critique panel beats self-correction (79.45% versus 47.95% on Penguins). It supplies the flat-panel baseline, but it has no editorial tier at all. Embodied LLM agents in organized teams shows that designated leadership improves team efficiency by 9.76% in household tasks. It suggests that structure matters in multi-agent systems, though only for physical tasks, never for evaluation.
The content of the flaw we target comes from the theory-validation literature. Constants are Critical in Regret Bounds shows that the asymptotically more sophisticated MVP algorithm consistently underperformed simpler UCBVI variants once implemented, so asymptotic claims are poor proxies for practical performance. Papers such as Shortest Paths in Weighted Unit-Disk Graphs, Linear Convergence with Condition Number Independent Access of Full Gradients, and Nibble local clustering leave practical efficiency unmeasured. They are legitimate as theory papers, but they show what a "practical claim, no evidence" pattern looks like. Our point is not that such papers should be rejected. It is that a board should flag when the practical language outruns the evidence.
The mechanism-design work in Impartial Peer Review and On Strategyproof Conference Peer Review constrains our design rather than supplying it. It shows that hard structural guarantees carry real costs, such as empty or undersized selections and lost comparability across clusters. That warns us that a gate can over-reject, so false alarms must be a primary metric.
Research Program
We build the evidence auditor as a separate agent. It receives the paper with styling removed and is restricted to two inputs: the paper's claims about practical or empirical performance, and the sections that could support those claims, namely experiments, tables, and appendices. It never sees the reviews, the rebuttal, or the abstract's framing. It outputs a short ledger that pairs each claim with its supporting evidence or with "none found." This isolation is deliberate. It keeps the auditor from being anchored by a confident review, and it removes the presentation cues behind the Presentation Trap.
We then wire the same ledger into the board in three ways. First, as an extra review that is averaged like any other. Second, as context handed to the meta-reviewer. Third, as a gate: an unsupported practical claim forces the area chair either to address it explicitly or to escalate the paper to a senior tier, which sees only the ledger and the contested section.
At every tier we log whether the planted flaw was mentioned. This lets us decompose each pipeline's failure into missed detection and lost propagation.
Baselines
The naive reference point reproduces the original pipeline from the perturbation study, with GPT-4o and Gemini 2.0 Flash as reviewers and meta-reviewer, using that study's prompts. It anchors our results to their reported 1.65-point Strong-Reject sensitivity. The file does not report a numeric soundness-propagation rate for this pipeline, so we will measure that rate ourselves.
The strongest existing structured competitor is PeerArg-style aspect aggregation. Its reported result is Macro F1 76.64% on PRA. We rerun it on our splits with the same backbone LLM in place of its extraction model, so that differences reflect aggregation rather than model strength.
We also run two hierarchy and panel baselines. One is the AgentReview hierarchy with an inclusive area chair. The other is the flat critique panel with majority vote from the multi-agent peer review work.
The key ablation is the auditor-as-extra-review condition. It uses identical auditor text, consumed by averaging. Gating "beats" the field if three conditions all hold. Flaw propagation, meaning a decision that drops a full tier or a score that falls by at least 1.5, must rise by at least 15 percentage points over the strongest baseline. False-alarm rate on unperturbed papers must rise by no more than 5 points. Decision Macro F1 on PRA and PeerRead must stay within 2 points of PeerArg's reported figure.
Every condition uses the same backbone models, the same papers, and a token budget capped at that of the most expensive baseline. Each condition gets the same number of prompt-tuning iterations, run on a held-out development split.
Why not just…?
Why not just add a checklist line to the meta-reviewer prompt ("does every practical claim have evidence?")? This is the most serious threat, and we run it as a baseline. If a single-prompt checklist matches the gated auditor on both propagation and false alarms, the structural claim is wrong and we will report that. Our prediction against it rests on the perturbation study's finding that meta-reviewers read the very sections they then misattribute. A checklist that still sees fluent prose and positive reviews should inherit the same anchoring.
Why not just use structured argument aggregation like PeerArg? PeerArg does beat end-to-end models on decision F1. But it resolves conflicting arguments by weighing them, so one well-founded soundness objection competes against several presentation compliments. Our per-tier logs will show whether PeerArg detects the flaw and then outweighs it, which is the propagation failure we predict.
Why not just rely on strategyproof or impartial selection mechanisms? Those mechanisms address strategic manipulation of scores among competing authors. They take reviews as given numbers, so they cannot recover a flaw that no score encodes, and they carry known costs such as empty selections. They are orthogonal to our question, and their costs warn us that gates must be checked for over-rejection.
Proposed Experiments
We use the 508-paper ICLR 2024 base from the perturbation study and add one new perturbation family, called claim-evidence mismatch. It comes in three variants: delete the experiments supporting a stated speedup or accuracy gain while keeping the claim; insert unsupported wall-clock or real-world-gain sentences into a paper's abstract and conclusion; and replace measured results with asymptotic statements only. Each perturbation has a matched control that edits presentation only, which lets us measure false alarms and Presentation-Trap misattribution.
To check that the gate does not damage ordinary decisions, we report Macro F1 on PRA and on PeerRead acceptance prediction. For the latter, PeerRead reports 65.3% accuracy on ICLR with surface features, which gives a floor any pipeline must clear. We also check robustness to the existing Strong-Reject flip and hostile-rebuttal edits, to confirm that a gate does not add a new single point of failure.
The decisive check is the ablation. If propagation rises when only the routing changes, structure is the cause. If propagation rises only when the auditor's own content improves, the structural claim fails. As a further check, we test whether the same gain appears across both backbone models and on the medical MOPRD reviews, which would separate a general property of boards from a quirk of one model.
References
- Aspect-Guided Multi-Level Perturbation Analysis of Large Language Models in Automated Peer Review
- AgentReview: Exploring Peer Review Dynamics with LLM Agents
- PeerArg: Argumentative Peer Review with LLMs
- Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- Position: Constants are Critical in Regret Bounds for Reinforcement Learning
- An Improved Algorithm for Shortest Paths in Weighted Unit-Disk Graphs
- Linear Convergence with Condition Number Independent Access of Full Gradients
- A Local Clustering Algorithm for Massive Graphs and Its Application to Nearly Linear Time Graph Partitioning
- Impartial Peer Review
- On Strategyproof Conference Peer Review
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications