When a group of language-model agents must reach a decision, such as a debate among answerers or a stack of peer reviews waiting for a meta-review, the usual fix for chaos is to appoint one central judge. That fix works, but it creates a single point of failure. A single point of failure is a component whose failure takes the whole system down with it. If the judge rewards fluent prose over correct reasoning, or caves to a confident wrong majority, every decision inherits that flaw. The obvious remedy is redundancy: replace one judge with several and vote. This note argues that the remedy is mostly an illusion. Several copies of the same model, reading the same inputs, fail on the same inputs. Network measurement has already seen this pattern. Millions of seemingly independent open DNS resolvers turn out to forward to a handful of upstream servers, so one upstream outage silences thousands of them at once (ODNS Clustering).
We propose to measure this hidden centralization in LLM judge panels directly. We then test whether an editorial-board structure removes it at matched cost. In this structure, associate editors see deliberately different views of the evidence, one of them is assigned to argue the minority case, and an editor-in-chief may act only on the specific points where the associate editors disagree. The quantity we care about is blast radius: how many final decisions flip when one component is corrupted. Clean-data accuracy is a secondary measure.
Core Claim
Two findings in the literature pull against each other. The first motivates centralization: adding a designated judge to multi-agent debate cut the disagreement collapse rate on CommonsenseQA from about 81% to 41% (Peacemaker or Troublemaker). Disagreement collapse is the share of cases where agents abandon a correct dissenting answer to match their peers. The second finding warns against relying on that judge. LLM judges of peer reviews reward analytical form, meaning length, vocabulary and layout, over epistemic function, meaning actual fault-finding. They also collapse multi-dimensional rubrics into one dominant factor (Articulate Intuition or Genuine Analysis?). Separately, the flip toward a perceived consensus is localized in middle transformer layers and is not reliably prevented by "be skeptical" prompts (Not Just RLHF).
Our claim is that these failures are input-triggered and shared across copies of a model. Replicating the judge therefore adds little protection against them. Protection has to come from structure. Different evaluators must receive different transformations of the evidence, so that a trigger present in one view is absent from another. A designated dissent role must keep minority arguments visible. The top-level authority must be gated so that, when it fails, the damage is confined to the cases where its subordinates disagree.
Blast radius makes this testable. Let $D(x)$ be the panel's decision on item $x$, and let $D_c(x)$ be its decision when component $c$ is corrupted. Corruption here means swapping in a biased model or injecting a fluent-but-wrong input. Then
$$B_c = \Pr_x\big[D_c(x) \neq D(x)\big],$$
and the worst-case blast radius is $\max_c B_c$ over all components. We predict two things. A homogeneous $k$-judge vote will have nearly the same worst-case blast radius as a single judge under input-triggered corruption. The view-diversified board will cut it substantially at a matched token budget.
Research Program
The board has three tiers. In the bottom tier, reviewers (or debaters) produce independent assessments. In the middle tier there are three associate editors, each built from the same base model so that any gain comes from structure rather than model diversity.
Each associate editor sees the same underlying evidence in a different form. The first reads a style-normalized rewrite of every review: each review is paraphrased to a fixed length and format, which borrows the idea behind the sanitized abstracts of BeerReview. The goal here is to remove the length and fluency cues that, according to Articulate Intuition or Genuine Analysis?, explain about 74% of the naive score advantage of AI reviews. The second reads the reviews as a structured graph of supporting and attacking arguments, organized by aspect, following the extraction stage of PeerArg. The third is the designated dissenter. It is shown the minority position and must write the strongest case for it, which makes the "single dissenting voice" that Not Just RLHF found protective into a fixed structural role. That paper explicitly leaves open where such a dissenter should sit inside an organizational hierarchy, and this design is one answer.
The top tier is the editor-in-chief. When all associate editors agree, their decision stands and the editor-in-chief cannot override it. When they disagree, the editor-in-chief receives only the contested point, the competing arguments, and the original passages they cite. It does not receive the full transcript, which avoids the lost-in-the-middle degradation documented on flat logs by Which Agent Causes Task Failures and When?.
This gating is the key to the single-point-of-failure argument. A corrupted editor-in-chief can only damage the disagreement set. A corrupted associate editor must also persuade the editor-in-chief before it can change an outcome.
We add one more mechanism as a separate, ablatable component. Each associate editor carries a trust score drawn from a Beta distribution with time decay, adapted from the per-node trust evaluation in DzTrust. In that system, the trust score of a compromised sensor node drops almost immediately. Here, an associate editor's score is updated whenever the editor-in-chief rules against it, and a low score lowers that editor's weight. DzTrust isolates compromised nodes in a sensor network. We ask whether the same bookkeeping can quarantine a corrupted evaluator. The two settings differ: an evaluator's errors are graded judgments rather than binary misbehavior, so we treat this as an empirical question.
The program has three analysis steps. First, we estimate pairwise error correlation between evaluators under each view, on clean and corrupted inputs. This tells us whether view diversity actually decorrelates failures. Second, we measure blast radius for every component of every architecture. Third, we keep the board's logs structured by tier, so that a failure can be traced to the tier that caused it. We check whether this makes attribution easier than on the flat traces of Who&When.
Baselines
The naive reference point is a single central judge that reads all reviewer outputs and decides. This is the configuration Peacemaker or Troublemaker showed beats leaderless debate. Every other number is read against it.
The flat decentralized reference is voting among equal agents without a judge, following Voting or Consensus?. That work reports 54.8% voting accuracy versus 28.4% for consensus on MuSR. We run its simple-voting protocol with the same base model.
The strongest competitor on our specific question is a homogeneous judge ensemble. It consists of $k$ copies of the central judge, each with sampling noise and a differently phrased prompt, combined by majority vote. We set $k$ so its total token count matches the board's average per item. It is the most direct way to remove the single point of failure without any of our structure, so if it matches the board on blast radius, our structural claim fails.
For the peer-review setting, the strongest existing aggregation method in the file is PeerArg (PeerArg). It reports Macro F1 of 76.64% on PRA (ICLR 2018) and 68.09% on MOPRD (medical journals), against an end-to-end LLM baseline at 72.5% and 59.5%. We rerun PeerArg on our splits rather than copying its numbers. On FIRSTPASS, the reference is the fine-tuned Qwen2.5-7B-Instruct reported at 80.5% accuracy and 78.2% F1-macro for predicting whether a manuscript needed standard or extended review (FIRSTPASS). We can reuse that model as a fixed reviewer-signal component, so that baseline and board differ only in aggregation.
The ablations isolate each ingredient. We give all three associate editors the identical raw view, which isolates view diversity. We let the editor-in-chief rule on every item, which isolates the gating rule. We replace the dissenter with a fourth ordinary associate editor, which isolates the dissent role. We turn off trust scoring.
All systems share the base model, the reviewer outputs, the prompt-tuning budget (the same number of development-set iterations), and the matched token budget. Beating the strongest competitor means two things together. On clean data, the board must stay within one point of the best baseline's accuracy or F1. Under corruption, it must reduce worst-case blast radius by at least half relative to the homogeneous ensemble, with non-overlapping bootstrap confidence intervals.
Why not just…?
Why not just use several independent judges and vote? This is the real threat, and it is why the homogeneous ensemble is our main baseline rather than a straw man. The hypothesis is that its errors are correlated: a padded, fluent wrong review fools every copy for the same reason. The evidence pointing this way is that length alone explains most of the judges' preference in Articulate Intuition or Genuine Analysis?, and that cue is identical for every copy. If our measured error correlation among ensemble members turns out to be low, the ensemble will win on cost and we will report that plainly.
Why not just hand everything to one frontier reasoning model? On structured multi-agent traces, models like o1 and DeepSeek R1 reached only 14.2% step-level attribution accuracy (Which Agent Causes Task Failures and When?). Frontier judges also collapse rubrics into a form-over-substance factor. More importantly, a single frontier model remains a single point of failure by construction: its blast radius under corruption is 100% of decisions. We include it as a baseline to measure how much clean accuracy it buys in exchange.
Why not just tell the judge to be skeptical? Not Just RLHF reports that defensive prompts overfit to the attack they were written for and fail when the framing of the pressure changes. A structural dissenter, by contrast, cut failure rates by 54 to 73 percentage points. We nonetheless run a skeptical-prompt judge under paraphrased corruption, so the comparison is made on our own data.
Why not just a better aggregator such as PeerArg? PeerArg improves clean F1, but it is a single fixed pipeline. A corrupted extraction step flows straight into its decision node. We evaluate its blast radius under the same injections. We also already reuse its argument graph as one associate editor's view, so a win for the board cannot be explained by having the graph.
Proposed Experiments
The first experiment uses clean ground truth. On CommonsenseQA and MuSR debates, we inject one confidently wrong debater and separately swap the judge for a version prompted to favor a fixed answer position. We measure accuracy, disagreement collapse rate, and blast radius for every architecture. This tests the structural claim where labels are unambiguous.
The second experiment uses peer review. On PRA, MOPRD and FIRSTPASS, we use three injections. We insert a length-padded, fluently written review that argues the wrong outcome. We corrupt one associate editor. We corrupt the editor-in-chief. The Articulate Intuition or Genuine Analysis? result that acceptance tier does not track reasoning quality warns that outcome labels are noisy. For that reason, we treat decision flips under injection as the primary measure, and F1 against labels as secondary. We also run every evaluator on the Kahneman4Review function-probe pairs, to check whether the style-normalized view actually removes the form-over-substance bias at the component level.
The decisive checks are three. If view-diversified associate editors show error correlation no lower than homogeneous copies, the mechanism is wrong. If the identical-view ablation matches the full board's blast radius, view diversity is not what helps. If a corrupted editor-in-chief's blast radius does not shrink to roughly the size of the disagreement set, the gating claim fails. If all three hold at matched token budget, we will have shown that the single point of failure in LLM judging is a property of shared inputs, not of judge count, and that an editorial board built around different views of the evidence removes it.
References
- ODNS Clustering: Unveiling Client-side Dependency in Open DNS Infrastructure
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy
- BeerReview: A Blockchain-enabled Peer Review Platform
- PeerArg: Argumentative Peer Review with LLMs
- Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- A distributed zero-trust scheme for airborne wireless sensor networks using dynamic identity authentication
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes