When several language models answer the same question, the simplest way to combine them is to take the most common answer. A more elaborate option is multi-agent debate, where the models read each other's reasoning and revise over several rounds. The evidence so far favors the simple option. Majority voting beats debate at matched compute. Debate adds no expected correctness under a clean theoretical model. Models that revise their own answers without outside feedback talk themselves out of correct answers more often than into them. Yet one diagnostic study finds that well-run debates rescue a wrong initial majority 74% of the time. This note proposes a way to reconcile the two findings and build on them. Debate should be treated as a perturbation applied to a vote. It should be run only on questions where the initial vote is fragile, meaning a small number of effectively independent agents switching would change the winner. A revision should be accepted only if it survives small, meaning-preserving changes to how peer arguments are shown, such as their order or labels.
The two ingredients come from different literatures. Fragility is the robustness-radius idea from social choice: the smallest input change that flips an outcome. It is corrected for the fact that agents built by the same model provider make correlated mistakes. The stability check tests whether a revision was driven by content or by presentation cues, and LLM judges are known to follow such cues without admitting it. The resulting system spends debate compute only where a correct answer is plausibly recoverable. It is never asked to overturn a robust majority, which is where self-correction does its damage.
Core Claim
Debate's net effect on accuracy has two parts that pull in opposite directions. It repairs some questions where the initial majority was wrong, and it breaks some where the majority was right. Averaged over a benchmark, the two roughly cancel or turn negative. Debate or Vote formalizes debate as Bayesian belief updating, proves it is a martingale (expected belief in the true answer does not rise across rounds), and reports zero-round majority voting at 0.7691 average accuracy against 0.7377 for two-round debate on Qwen2.5-7B across seven benchmarks. Large Language Models Cannot Self-Correct Reasoning Yet shows the harm side directly: GPT-4 on GSM8K falls from 95.5% to 89.0% under intrinsic self-correction, and correct-to-incorrect flips outnumber repairs. The repair side is visible in A Diagnostic Study of Multi-Agent LLMs for Real-World Debates, where high-process debates recover from incorrect initial majorities 74% of the time versus 40% for low-process ones.
Our claim is that the two parts can be separated before debate runs, using only unlabeled signals.
- Fragility. A dependence-corrected robustness radius of the round-zero vote predicts which side a question falls on. Robust majorities are almost always right, so debating them can only hurt. Fragile majorities are where recovery happens.
- Stability. Within the fragile set, a debate-induced change of answer is trustworthy only if it recurs when the peer transcript is shown in a different order or with provenance labels swapped. A change that disappears under that perturbation is a shortcut-driven flip of the kind documented in The Silent Judge. That paper finds judges shift verdicts by up to +30% on ELI5 under recency cues and never acknowledge the cue (0% acknowledgment rate).
The martingale result constrains this claim; it does not refute it. The proof concerns expected belief. Accuracy is the argmax of belief, so it can move even when expected belief does not. Real models may also update asymmetrically, finding correct arguments more persuasive than incorrect ones, which the 74% recovery figure suggests. If debate is an exact martingale even conditional on fragility, gated debate will not beat voting. The experiments below are designed so that this outcome would be visible and reported.
Research Program
Measuring fragility. For a panel of $K$ agents with round-zero answers, let $w$ be the plurality winner and $r$ the runner-up. Following the robustness-radius notion from Robustness of Greedy Approval Rules, define the fragility radius as the smallest total weight of agents that must switch from $w$ to $r$ to change the winner:
$$\rho = \min_{S \subseteq \{i : a_i = w\}} \sum_{i \in S} v_i \quad \text{subject to the winner changing once all agents in } S \text{ vote } r,$$
Here $a_i$ is agent $i$'s answer and $v_i \in (0,1]$ is its effective independence weight. For equal weights, $\rho$ is about half the vote margin.
The weights are the new part. We fit a class-dependent Ising model to the panel's votes on an unlabeled pool of questions, using the unsupervised procedure of Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models. That paper shows correlated judges make majority voting confidently wrong, and that coupling is strongest within a provider family. We then set $v_i$ to shrink with agent $i$'s coupling to the rest of the panel. As a result, three agreeing agents from one family count for less than three agreeing agents from three families.
Two notes on this step:
- The margin-versus-robustness warning in On the Robustness of Winners: Counting Briberies in Elections motivates using a radius instead of a raw lead.
- Exact robustness computation is intractable for general voting rules (Robustness of approval-based multiwinner voting rules). That hardness does not bite here, because the answer space is small and the rule is weighted plurality, so $\rho$ reduces to a sorted greedy sum.
Gating. Questions with $\rho$ above a threshold $\theta$ return the round-zero plurality answer immediately. Fragile questions go to debate. The debate protocol uses independent drafts before agents see peers, taken from Voting or Consensus?, where this All-Agents Drafting step is shown to curb early sycophancy. Debate is capped at two rounds, because the same paper documents accuracy drift in longer debates. $\theta$ is chosen on a held-out labeled split to maximize accuracy at a fixed token budget.
Stability filter. For each fragile question we run $M$ debate replicates (default $M=2$). Each shows the identical peer content in a permuted order, with model-identity labels either randomized or removed. A changed answer is accepted only if a strict majority of replicates move to the same new answer. Otherwise the system keeps the round-zero plurality answer or, in the selective variant, abstains.
Abstention follows the logic of Selective Preference Aggregation, which declines to rank items when evaluators disagree too much, and of Trust or Escalate, which abstains or escalates when simulated-annotator agreement is low. It extends both from pairwise or static ballots to answers produced by an interacting panel.
Compute accounting. The gate saves the debate cost on robust questions. That saving pays for the $M$ replicates on fragile ones. All comparisons are therefore made at matched total generated tokens, and we report the fraction of questions debated.
Analysis. The main analytical product is a decomposition of every method's accuracy change relative to round-zero voting into repairs (wrong to right) and breakages (right to wrong). Each is plotted as a function of $\rho$. This decomposition directly tests whether the martingale's aggregate zero hides a positive region at low $\rho$ and a negative region at high $\rho$.
Baselines
Round-zero majority voting (the "do nothing" reference). Each agent answers once and the plurality answer is taken. Under homogeneous sampling this is self-consistency. It is the strongest simple competitor because both prior comparisons found it hard to beat. Large Language Models Cannot Self-Correct Reasoning Yet reports 88.2% on GSM8K for self-consistency with nine responses against 83.0% for debate with nine responses. Debate or Vote reports 0.7691 average accuracy on Qwen2.5-7B. We give voting extra samples until its token count matches the gated system's total, so it has every advantage of scale. Beating it means higher accuracy at equal tokens on at least the reasoning-heavy benchmarks, with the gain confidence interval excluding zero across three seeds.
Always-debate. Every question is debated with the same drafting and two-round protocol, but without a gate or filter. This isolates what gating adds. The reference point is the 0.7377 two-round decentralized debate figure from Debate or Vote.
MAD-Conformist. From Debate or Vote, this method biases debating agents toward the early majority. The paper reports it raises two-round debate from 0.7332 to 0.7625 on Qwen2.5-7B. It is the most direct existing competitor, because it also tries to stop debate from overturning good majorities, but it does so uniformly rather than per question. We run it on the same panels, prompts, and token budget. Beating it means higher accuracy and a lower breakage rate, without a lower repair rate.
SID. SID: Multi-LLM Debate Driven by Self Signals stops debate early using token-level logits and compresses context using attention maps. It reports about 44% versus about 39.5% for standard debate on MMLU-Pro with LLaMA-3.1-8B, with up to 40% fewer tokens. It is the strongest compute-saving debate method in the file. It requires white-box access, so we compare only on open-weight panels.
Ablations. Three ablations isolate the key ingredients:
- Raw-margin gate. Equal weights $v_i = 1$ and no stability filter. This isolates the dependence correction.
- Fragility gate without the stability filter. This isolates the replicate check.
- Stability filter applied to all questions without gating. This tests whether the filter alone accounts for the gains.
Why not just…?
Why not just gate on the raw vote margin or answer entropy? This is the most serious threat. On a homogeneous panel of samples from one model, the Ising couplings are roughly uniform, the weights become nearly equal, and $\rho$ collapses to the raw margin. Any advantage there would have to come from the stability filter alone. We therefore expect no difference on homogeneous panels and say so up front. The dependence correction earns its place only on mixed-family panels. There, Dependence-Aware Label Aggregation shows correlated judges inflate apparent agreement, reporting gains of 6.4 to 9.8 points over conditionally independent weighted voting. The raw-margin ablation on heterogeneous panels settles the question. If it matches the dependence-weighted gate there, we drop the Ising step and report the simpler gate as the result.
Why not just aggregate the round-zero votes better and skip debate entirely? Ising aggregation of the zero-round votes is cheap and may capture most of the gain, since it already discounts redundant agents. We run it as a baseline. Better aggregation can only reweight answers that some agent already produced. It cannot recover a correct answer that no agent, or only a small minority, gave initially. The 74% recovery figure in A Diagnostic Study concerns exactly those cases. The repair decomposition shows directly whether debate on fragile questions produces correct answers absent or in the minority at round zero. If it does not, aggregation alone wins and debate is unnecessary.
Why not just write a better initial prompt? Large Language Models Cannot Self-Correct Reasoning Yet and Debate or Vote both show that apparent refinement gains often reflect weak first prompts or answer-parsing failures. We adopt this criticism rather than argue with it. Every method uses the same strongly specified prompt and a strict answer format, and parsing failures are reported separately. The proposal claims gains only beyond a well-prompted single turn and well-prompted voting.
Why not just use a frontier model as the final arbiter of fragile cases? A single strong judge is a natural escalation target, and we include it as an alternative resolver of fragile questions. However, The Silent Judge shows such judges follow presentation cues without acknowledging them. Trust or Escalate notes that standalone GPT-4 rarely reaches 80% human agreement on Chatbot Arena. We therefore apply the same order-permutation test to the arbiter and report how often its verdict changes. If it is both accurate and stable, it is a legitimate cheaper alternative to the debate stage.
Proposed Experiments and Decisive Checks
Benchmarks. We evaluate on GSM8K, CommonSenseQA, MMLU-Pro, GPQA, MuSR, and StrategyQA. This spans the knowledge-heavy tasks where Voting or Consensus? found consensus-style filtering helps (54.0% versus 51.7% on MMLU) and the reasoning tasks where voting dominates (54.8% versus 28.4% on MuSR).
Panels. Two panels are run throughout:
- Homogeneous: five samples of Qwen2.5-7B, which aligns with the Debate or Vote numbers.
- Heterogeneous: open-weight models from at least three families, with two members from one family so that correlated agreement is deliberately present.
Ising couplings are fit on unlabeled questions disjoint from the test sets. The threshold $\theta$ and replicate count $M$ are tuned on a labeled validation split, with the same tuning budget given to every baseline's hyperparameters.
Metrics. We report:
- accuracy at matched generated tokens;
- repair and breakage rates relative to round zero;
- the fraction of questions debated;
- selective accuracy versus coverage for the abstaining variant.
Where the file gives no per-benchmark numbers for some datasets (for example GPQA, StrategyQA, and HotpotQA; no reported result found on Lacuna for per-benchmark debate-versus-voting accuracy there), we produce our own voting and debate baselines rather than cite them.
Check 1: the decomposition. Plotting repairs and breakages against $\rho$ must show breakages concentrated at high $\rho$ and repairs at low $\rho$. If debate's net effect is flat or negative in every $\rho$ bucket, the martingale picture holds conditionally and the proposal fails at its foundation.
Check 2: matched-compute accuracy. The gated, stability-filtered system must beat compute-matched round-zero voting and MAD-Conformist on the reasoning benchmarks. Matching voting on knowledge tasks while debating only a minority of questions would still be a partial win in cost.
Check 3: the stability filter. The filter must cut breakages among fragile questions substantially while keeping most repairs. A useful diagnostic is how often an unfiltered answer change reverses under a permuted transcript. This measures, for the first time, how much of debate's harm is driven by presentation cues.
Check 4: the dependence correction. On the heterogeneous panel, the dependence-weighted gate must outperform the raw-margin gate. On the homogeneous panel, the two should tie. A tie on both panels means the Ising step is removed from the method.
References
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
- Large Language Models Cannot Self-Correct Reasoning Yet
- A Diagnostic Study of Multi-Agent LLMs for Real-World Debates
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Robustness of Greedy Approval Rules
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
- On the Robustness of Winners: Counting Briberies in Elections
- Robustness of approval-based multiwinner voting rules
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Selective Preference Aggregation
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
- SID: Multi-LLM Debate Driven by Self Signals