When several language models check each other's answers, the failure that matters most is a correct model changing its answer because a confident peer disagreed, not a model being wrong. Multi-agent debate studies find that this kind of social capitulation grows with every round. Today's fixes are soft: a judge prompt, a persona told to be less agreeable, a fixed limit on rounds. We propose taking a mechanism from academic journals and making it hard. On an editorial board, a referee cannot simply announce that an author should change a conclusion. The objection must point at something specific, and disputes that referees cannot settle go up to an associate editor. We encode those procedures as event-condition-action (ECA) rules, the "when this happens, if this holds, then do that" triggers used in active databases and middleware. The rules run in a small deterministic engine that sits between the models. It does not live inside any model's prompt.
The core claim is that one conditional rule, which we call the revision guard, removes most sycophantic answer flips while keeping the useful ones. The rule says a model may change its final answer only if a reviewer has flagged a specific step of its solution and the revision actually changes that step. The guard must be enforced outside the models. Models are known to obey "never do X" far better than "don't do X if Y" (SENTINEL), so writing the same rule into the prompt should not be enough. The editorial hierarchy supplies a second rule: escalate to a single adjudicator only when reviewers split with high confidence. This makes arbitration happen when an event calls for it, instead of on every question.
Core Claim and What Is New
Three pieces of prior work motivate the design. Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration runs a create–review–revise loop with confidence-scored critiques. It shows that external critique prevents the self-correction collapse in which one model talks itself out of a right answer. It is flat and synchronous, and the reviser may change its answer for any reason. Peacemaker or Troublemaker measures sycophancy in debate and finds it intensifies across rounds. In its data, an agent's sycophancy score tracks how often it abandons correct answers. A centralized judge roughly halves disagreement collapse. However, that judge reads raw transcripts and acts only as a final summarizer. Preventing Rogue Agents Improves Multi-Agent Collaboration behaves like an ECA system: a trained classifier over token-uncertainty statistics fires rollbacks when it predicts failure. Its triggers are statistical rather than declarative, and it has no notion of who is entitled to change what.
Our proposal differs from all three. We gate the act of revision itself, using a declarative condition over structured review records. We also trigger hierarchical arbitration by an explicit event rather than running it always or never. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that hierarchy helps LLM teams, but its organizational rules are prompts rewritten by a critic model. The authors themselves note that behavior under these rules is hard to bound. We test directly whether moving such rules from prompt to engine changes behavior.
Research Program
The system has three models acting as authors and reviewers, one associate-editor model that sleeps by default, and a rule engine of a few hundred lines. Each model first writes a numbered step-by-step solution. Each model then reviews every peer, but it does not return free prose. It returns a structured record: which peer, agree or disagree, the index of the first disputed step, the value it believes is correct at that step, and a confidence from 1 to 10. A one-sentence justification is attached as an optional field. The design of the channel follows Communication Boundary Control for Safer Multi-Agent Language Agents, which argues that decisions should be made over structured records before any language is realized, because free wording leaks pressure that the record does not carry.
The engine listens for three events: a review was submitted, a revision was submitted, and a round ended. Four rules respond. The first is a desk-accept rule: if all records agree, stop immediately. The second is the revision guard: a revision that changes the final answer is admitted only if some record disputed a specific step of that solution and the revised solution differs at that step. Otherwise the engine silently reverts the final answer to its previous value. The third is an escalation rule: if verdicts split and the most confident dissenter reports confidence at or above a threshold $\tau$, wake the associate editor. The editor sees only the disputed step and the competing values, so it rules on one local claim rather than re-solving the whole problem. This is a modest borrowing from the policy-projection idea in Decentralized Distributed Data Usage Control, where global rules are checked locally when local information suffices. The fourth is a hard cap of three rounds, in line with the debate-length finding of Yao et al.
The analytic object is the quality of answer changes. For a run, let $p_{\uparrow}$ be the fraction of agent-rounds in which a wrong answer becomes right and $p_{\downarrow}$ the fraction in which a right answer becomes wrong. We define net flip gain as
$$G = p_{\uparrow} - p_{\downarrow}.$$
The revision guard should raise $G$ mainly by cutting $p_{\downarrow}$, which corresponds to the negative agreement rate of Yao et al. It must not cut $p_{\uparrow}$ by much. Accuracy, token cost, disagreement collapse rate, and the sycophancy score from Yao et al. are reported alongside $G$. So is a rule-violation rate: how often an agent attempts an unsupported flip that the engine blocks.
Baselines
The naive reference point is majority vote over independent samples (self-consistency) at the same total token budget as our full system, with no reviewing at all. If rule-gated review cannot beat plain sampling at matched cost, the editorial machinery adds nothing.
The second reference is single-agent self-correction. On Colored Objects it is reported to drop accuracy from 66% to 28% (Xu et al.), so it serves as a floor that shows the collapse we aim to prevent.
The strongest existing method on our question has two parts, and we run both. The first is the flat create–review–revise protocol of Xu et al., whose strongest reported number in the file is 79.45% accuracy on Penguins against 47.95% for self-correction. The second is judge-centralized debate from Yao et al., whose reported number is a reduction in disagreement collapse on CommonsenseQA from about 81% to 41%. We reimplement both with the same open models Yao et al. used (Qwen3-32B and LLaMA 3.3-70B Instruct), the same three agents, the same round cap, and the same per-method prompt-tuning budget on a held-out development split. Beating them means two things. First, equal or higher accuracy at equal or lower tokens. Second, a lower negative agreement rate and higher $G$, both with confidence intervals that exclude the baseline across both models.
We also adapt the learned trigger of Barbi et al. as an alternative escalation signal. Its uncertainty classifier decides when to wake the associate editor, in place of our confidence-split rule. No reported result for this method on our benchmarks was found on Lacuna, so it is a new comparison.
Two ablations isolate the key ingredient. In the soft-rule ablation, the identical four rules are written into every agent's and the editor's prompt, and no engine enforces them. In the guard-off ablation, the engine runs escalation and stopping but admits any revision. A third ablation replaces the structured review channel with free prose to test the leakage argument.
Why not just…?
Why not just spend the same tokens on more samples and a majority vote? For accuracy alone, this is a real threat. Adaptive self-consistency with early stopping on agreement may match us on easy arithmetic sets. The matched-budget comparison settles it. We will report accuracy-versus-tokens curves per dataset and accept the result if sampling dominates. Our claim survives only where the guard adds something sampling cannot: correcting a minority-right answer through a step-specific objection. This should show up on symbolic sets such as Penguins and Colored Objects, where Xu et al. found that critique matters most.
Why not just tell agents to be less agreeable? Yao et al.'s persona grid shows agreeability can be dialed down. However, a stubborn agent also refuses correct critique, so it lowers $p_{\downarrow}$ by also lowering $p_{\uparrow}$. We run the low-agreeability personas as a baseline and compare $G$. The guard is meant to be selective where a persona is uniform, and if a persona reaches the same $G$ the guard is unnecessary.
Why not just write the rules into the prompt or the judge's instructions? This is the hypothesis our soft-rule ablation exists to test. SENTINEL's finding that models handle conditional prohibitions poorly predicts that soft rules will be violated often. The violation rate measures this directly. If prompt-stated rules achieve the same negative agreement rate as engine-enforced ones, the engine is unnecessary and we will say so.
Why not just always call the judge? An always-on judge is Yao et al.'s centralized condition. It costs an extra call per question and lets the judge's own sycophancy toward confident transcripts act on every item. Event-triggered escalation should match its accuracy at lower cost. The comparison at matched budget shows whether that holds.
Proposed Experiments
We evaluate on MMLU Pro and CommonsenseQA, where Yao et al. measured sycophancy, and on GSM8K, SVAMP, ARC-c, StrategyQA, Colored Objects, and Penguins, where Xu et al. measured peer review. Both open models are used throughout, with three seeds per configuration.
The first decisive check is a planted-dissenter test. One of the three agents is instructed to argue a wrong answer with confidence 9, and we count how often correct agents flip. Without the guard, the flip rate should resemble Yao et al.'s collapse figures. With the engine-enforced guard it should fall sharply, while the soft-rule version should fall less.
The second check is the reverse. We plant one wrong agent among two correct reviewers and measure whether it is still corrected. A guard that blocks this recovery fails, whatever it does for negative agreement.
The third check sweeps the escalation threshold $\tau$ and traces accuracy against associate-editor invocations. The goal is a regime where the editor is woken on a small fraction of items yet recovers most of the always-on judge's gain.
We deliberately do not attempt long-horizon settings such as PaperBench or ReviewBench from ResearchTown. The guard needs answers with checkable steps, and extending it to open-ended review is a later question that this study's results should justify or rule out.
References
- SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of LLM-based Embodied Agents
- Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- Preventing Rogue Agents Improves Multi-Agent Collaboration
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- Communication Boundary Control for Safer Multi-Agent Language Agents
- Decentralized Distributed Data Usage Control
- PaperBench: Evaluating AI’s Ability to Replicate AI Research
- ResearchTown: Simulator of Human Research Community