When several language-model agents disagree, the simplest way to pick an answer is to let them vote. Recent controlled studies find that voting is hard to beat, and that letting agents talk it over often makes things worse. The failure case is specific and well documented. Sometimes a minority holds the correct answer together with a valid reason for it. Free-text discussion does not reliably reward that reason. Minority agents give in to the majority, and voting simply counts heads. We propose a different kind of feedback between rounds, borrowed from the Delphi method, the expert-panel protocol in which panelists answer anonymously and independently, then see a controlled summary of the group's views before revising. In our version the summary is a scored argument graph rather than a median rating or a pile of quoted rationales.
Concretely, each agent's rationale is broken into atomic claims. Claims that two agents both made are merged into a single node, so they count once. The claims are linked by support and attack edges, forming a bipolar argumentation framework (a graph in which arguments can reinforce as well as undermine each other). Each candidate answer is then scored with a gradual semantics, a deterministic rule that computes an argument's strength from the strengths of its supporters and attackers. Agents see which answers survive scrutiny and which arguments against their own answer still stand unrebutted. Because duplicate claims are merged, an answer's strength depends on what is argued, not on how many agents argued it. The core claim is testable: argument-graph feedback should rescue correct minority positions more often than voting, free debate, or Delphi with ordinary rationale feedback, and it should not lose accuracy on average.
Where this sits
Three bodies of work motivate the gap.
The first is the multi-agent debate literature, which documents the failure we target. Debate or Vote models unstructured debate as a martingale, a process whose expected correctness does not drift over rounds. On Qwen2.5-7B it reports zero-round majority voting at 0.7691 average accuracy against 0.7377 for two-round decentralized debate. It argues that debate helps only once some bias toward truth is injected. Can LLM Agents Really Debate? finds two things on Knight–Knave–Spy puzzles: majority agents keep their stance over 97% of the time, and correction exceeds 90% when reasoning is formally valid. Peacemaker or Troublemaker ties sycophancy to abandoning correct answers (r = 0.902). Voting or Consensus? shows consensus protocols collapsing to 28.4% on MuSR, against 54.8% for voting.
None of these papers represents the arguments being exchanged. Our hypothesis is that gradual semantics over a deduplicated graph is exactly the kind of truth-ward bias the martingale analysis calls for: a valid argument that nobody rebuts keeps its strength, however few agents voiced it.
The second line is Delphi, which supplies the protocol. The Human-AI Hybrid Delphi Model reports 95% replication of historical consensus items. It names "interpretive flattening" as the core weakness of median-based feedback, but its fix depends on a human facilitator who codes rationales into flat themes. We keep the Delphi elements, namely anonymity, independent first drafts, and controlled feedback. We replace the facilitator's thematic coding with an explicit graph that records which reason undercuts which.
The third line is formal argumentation, which supplies the method and also constrains it. Adapting the DF-QuAD Algorithm to Bipolar Argumentation gives a semantics that handles cycles. The axioms of Amgoud et al. motivate our design choice that quality should beat quantity. Aggregating bipolar opinions through bipolar assumption-based argumentation is the closest prior work. It aggregates agents' bipolar opinions with discrete quota rules and proves that only dictatorship preserves coherence once three or more assumptions are involved. That work is one-shot, uses accept/reject labels, and is illustrated on a seven-person case. We instead aggregate continuously and iteratively, and we test at benchmark scale.
The constraint comes from Mining Bipolar Argumentation Frameworks from Natural Language Text, which reports that extracting graphs from text is noisy and brittle. Noisy edges are therefore the main risk to the proposal. We measure edge accuracy directly rather than assuming it.
Core Claim
Feedback computed from a deduplicated support/attack graph scored by gradual semantics shifts a deliberating panel toward answers backed by unrebutted arguments. The prediction has three parts. The rate at which a correct minority wins should rise relative to voting and free debate. Average accuracy should be at least that of majority voting. Agents abandoning correct answers should become less common. If Delphi with plain rationale feedback does just as well, the graph is not doing the work, and the claim fails.
Research Program
The system runs a small number of rounds with $k$ agents, which in the main setting are five instances of the same model.
In round zero, each agent answers independently and writes a rationale, never seeing its peers. A coordinator model then rewrites every rationale into atomic claims. It merges claims that are equivalent across agents and keeps a provenance count for reporting only. It labels each ordered pair of claims, and each claim–answer pair, as support, attack, or neither.
Every claim starts at a uniform base score $\tau(a)=0.5$, deliberately ignoring which agent said it. We do this so that strength cannot be inherited from confidence or majority. A variant that sets base scores from agent confidence is run as an ablation.
Strengths are computed with DF-QuAD. Let $\sigma(b)$ be the current strength of argument $b$. The combined force of a set of arguments is $v = 1-\prod_b (1-\sigma(b))$. With $v_s$ the force of the supporters of $a$ and $v_t$ the force of its attackers,
$$\sigma(a) = \tau(a) + (1-\tau(a))(v_s - v_t) \ \text{if } v_s \ge v_t, \qquad \sigma(a) = \tau(a) - \tau(a)(v_t - v_s) \ \text{otherwise},$$
and this update is iterated to a fixed point on cyclic graphs.
Each agent then receives the Delphi statistic, meaning the vote distribution, together with three pieces of graph feedback: the strength of every candidate answer, the strongest standing attacks on its own answer, and any of its own arguments that were attacked. Agents may revise their answer, but they may contribute only new claims; restating an existing claim adds nothing to the graph. The panel stops when answers and strengths are stable or after four rounds. The final decision is reported two ways, as the final-round vote and as the answer with the highest strength.
The graph also lets us examine the remaining open questions directly. We measure edge accuracy against ground truth where it exists. We track how often the aggregated graph contains the configurations behind Dickie et al.'s impossibility result. We compute dialectical faithfulness in the sense of Kotonya and Toni, asking whether answer strength tracks correctness, at a scale that paper deferred to future work.
Baselines
The naive reference points are a single agent and zero-round majority voting over the same five independent drafts. Majority voting is the strongest simple competitor, because Debate or Vote shows it matching or beating debate; that paper reports 0.7691 average accuracy for Qwen2.5-7B. We rerun it ourselves with the same models, sampling settings, and benchmark splits. The proposal beats it only if average accuracy is statistically no worse, the correct-minority win rate is higher, and both hold under a paired bootstrap across items.
The strongest existing debate methods are three. The first is decentralized debate with MAD-Conformist anchoring, which raised two-round debate on Qwen2.5-7B from 0.7332 to 0.7625 in Debate or Vote. The second is consensus and voting protocols with All-Agents Drafting, reported as a 3.3% gain in Voting or Consensus?; on MuSR, the voting protocol's 54.8% is the reference number from that paper. The third is WISE-style Dawid-Skene aggregation, an unsupervised model that learns per-agent reliability from agreement patterns. WISE reports it raising GPT-4o from 39.1% to 52.6% on EvoChart-QA. It has no reported result on our text benchmarks, so we run it ourselves. All of these baselines receive the same model, the same number of agents, the same maximum number of rounds, and a matched token budget. Each gets the same small prompt-tuning budget on a held-out dev split.
The ablations isolate the key ingredient. The most important is Delphi with flat rationale feedback: identical anonymity, drafting, and rounds, but agents receive the raw anonymized rationales plus the vote distribution and no graph. This is the facilitator-free analogue of HAH-Delphi. A second ablation gives the same extracted graph to a language-model judge in place of the DF-QuAD computation, which tests whether the deterministic semantics matter. A third keeps duplicate claims as separate nodes, which lets headcount back in. A fourth sets base scores from agent confidence. A fifth uses oracle edges on the logic puzzles, which separates extraction error from the value of the mechanism itself.
Why not just…?
Why not just take a majority vote? Voting is a genuine threat on average accuracy, and we may not beat it overall. Its failure is structural: it cannot recover when the majority is wrong. Can LLM Agents Really Debate? shows that wrong majorities are sticky. For that reason we stratify every result by whether the initial vote was correct, and we pre-register the correct-minority subset as the primary endpoint. If argument-graph feedback does not raise wins on that subset, the proposal fails.
Why not just run Delphi-style debate with anonymity and independent drafts? This is the most dangerous alternative. Much of the benefit could come from the Delphi elements alone, since drafting by itself helped in Voting or Consensus?. The flat-rationale ablation answers this directly. Our claim requires the graph condition to beat it on the correct-minority subset and on the rate of abandoning correct answers.
Why not just have a strong model read all the rationales and judge? A capable judge may implicitly do what DF-QuAD does. However, centralized judging hurt accuracy in Debate or Vote. The graph-to-judge ablation shows whether an explicit, inspectable rule adds anything beyond giving the judge structure.
Why not just learn agent reliability with Dawid-Skene? That method weights agents by how often they agree with others, so it inherits majority bias on hard items. It also cannot inspect reasons. Running WISE-style aggregation on identical drafts settles whether reliability weighting already captures the gain.
Proposed Experiments
The primary testbed is the Knight–Knave–Spy benchmark from Can LLM Agents Really Debate?: 1,800 puzzles of sizes 4 to 9. That paper reports debate raising accuracy from 17.33% to 69.33% at size 4. Every claim of the form "if A is a knight then B lies" can be checked mechanically, so edge accuracy and argument validity are directly observable. The oracle-edge ablation also lives here.
The second testbed is MuSR, 756 narrative reasoning items on which consensus collapses. CommonsenseQA and MMLU-Pro provide comparisons with the sycophancy and voting studies, using Qwen2.5-7B, Llama-3.1-8B, and one larger model.
Metrics are accuracy, correct-minority win rate, negative agreement rate, rounds to stability, tokens per item, and calibration of the highest strength against correctness, which is the faithfulness check.
As a separate test of the edge labeler, we score Feedback Prize segments with DF-QuAD over model-labeled edges and compare against the similarity-edge sp-ranking of Barile and d'Amato, which reports AdaBoost log loss improving from 1.0855 to 1.0844.
Three checks are decisive. First, if correct-minority wins do not exceed both voting and flat-rationale Delphi, the mechanism fails. Second, if oracle edges help but extracted edges do not, the bottleneck is extraction, not the semantics. Third, if counting duplicates as separate nodes performs as well as merging them, then quality-over-quantity is not what drives any gain.
References
- Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?
- Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- The Human-AI Hybrid Delphi Model: A Structured Framework for Context-Rich, Expert Consensus in Complex Domains
- Adapting the DF-QuAD Algorithm to Bipolar Argumentation
- Evaluation of Arguments from Support Relations: Axioms and Semantics
- Aggregating bipolar opinions through bipolar assumption-based argumentation
- Mining Bipolar Argumentation Frameworks from Natural Language Text
- Towards a Framework for Evaluating Explanations in Automated Fact Verification
- WISE: Weighted Iterative Society-of-Experts for Multimodal Multi-Agent Debate with Probabilistic Consensus
- MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
- Argumentation Ranking Semantics as a Feature for Classification - On Automatic Evaluation of Argumentative Essays