When several language models debate a question, the debate usually ends in a crude way. Either the agents vote, or they are pushed toward agreement, or a separate model reads the transcript and announces a winner. None of these steps looks at the structure of the argument. A correct dissenter who refutes the majority's key premise counts as one vote, the same as any other agent. A fluent but unanswered error counts the same as one that was knocked down two turns earlier. This proposal treats the finished transcript as a text to be parsed. A small learned scorer reads the whole debate and predicts, for every ordered pair of claims, whether claim $i$ attacks claim $j$. A soft version of formal argumentation semantics then asks which claims survive unrefuted. The answer backed by the strongest surviving claim wins.
The scorer is a biaffine parser, a pairwise scoring layer from dependency parsing that rates directed "A governs B" relations. It gives each token a separate "head" view and "dependent" view and combines them with a bilinear term plus per-node prior terms. Biaffine parsers already recover non-tree argument graphs from monologue text. What is new here is training one end to end, from nothing but whether the debate's chosen answer was right, so that its predicted attack graph decides LLM debates. In two sentences: debates should be won by the arguments that survive, not by headcount. We can learn to find those arguments cheaply with a directed pairwise parser trained through a differentiable "which claims are still standing" computation.
Core Claim
Kaesberg et al. (Voting or Consensus? Decision-Making in Multi-Agent Debate) found that how a debate is decided matters a great deal, and that the best rule depends on the task. On knowledge tasks, requiring consensus beats voting by 2.8 points on average, because the agreement requirement filters out hallucinations. On reasoning tasks, voting beats consensus by 13.2 points on average. On MuSR murder mysteries the gap is 54.8% versus 28.4%, because pressure to agree makes agents abandon sound reasoning chains. We claim that adjudicating over a predicted attack graph gets the benefit of both rules. Agents never have to converge, so sound minority reasoning is preserved. Yet an answer only wins if its support survives the other agents' attacks, so shared but refuted errors are filtered. The claim is falsifiable on that same split. If graph adjudication does not beat voting on knowledge tasks and consensus on reasoning tasks within a single setup, the claim fails.
The closest prior work plays three distinct roles. Ruiz-Dolz et al. (Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks) motivate the approach. They combined formal acceptability semantics with a graph network to predict winners of professional human debates, reaching 0.64 weighted F1 against 0.33 for a flat Longformer reader. However, they used hand-annotated argument graphs and evaluated offline. We predict the graph and apply it to machine debates with checkable answers.
Morio et al. (Towards Better Non-Tree Argument Mining: Proposition-Level Biaffine Parsing with Task-Specific Parameterization) and BiPAM (End-to-End Argument Mining as Biaffine Dependency Parsing) supply the method. They score directed support relations between propositions in single-author text, with separate parameters for node typing and edge scoring. We reuse proposition-level biaffine scoring but apply it to multi-author dialogue. We also supervise it through the decision it induces rather than only through edge labels.
Debate-side methods that filter content set the bar we must clear. Memory masking (Multi-Agent Debate with Memory Masking) judges each utterance alone, by perplexity or by asking an agent to tag it. Masked Agent Collaboration (Adaptive Cluster Collaborativeness Boosts LLMs Medical Decision Support Capacity) drops the agent least similar to its peers. That rule can, by construction, remove a correct dissenter. Neither method represents the fact that B attacking A is not the same as A attacking B.
Method
Agents debate in a light structured format. Each turn, an agent states a current answer and a short list of numbered claims. This format removes the segmentation problem that BiPAM had to solve, and it fixes which answer each claim supports. Only the relations between claims must be learned.
The whole transcript is encoded once by a mid-sized language model. Each claim becomes a node vector by pooling over its span, together with embeddings for the agent who made it and the round it appeared in. Following Simpler but More Accurate Semantic Dependency Parsing, two small networks project each claim vector $\mathbf{h}$ into an attacker view $\mathbf{h}^{\text{att}}$ and a target view $\mathbf{h}^{\text{tgt}}$. The probability that claim $i$ attacks claim $j$ is
$$p_{i\to j}=\sigma\!\left((\mathbf{h}_i^{\text{att}})^\top \mathbf{W}\,\mathbf{h}_j^{\text{tgt}}+\mathbf{u}^\top\mathbf{h}_i^{\text{att}}+\mathbf{v}^\top\mathbf{h}_j^{\text{tgt}}+b\right),$$
Here $\mathbf{W}$ captures how well a particular attacker fits a particular target. The term $\mathbf{u}$ is a per-claim "this claim's objections tend to land" prior. The term $\mathbf{v}$ is a per-claim "this claim is fragile" prior, and $b$ sets the overall density of attacks. A separate biaffine label head, as in Dozat and Manning's factorization into edge and label prediction, classifies support versus attack. The core decision uses attack edges only.
Acceptability is computed as a soft relaxation of grounded semantics. Every claim starts out accepted, $a_j^{(0)}=1$, and the update is iterated for a fixed number of steps $T$:
$$a_j^{(t+1)}=\prod_{i\neq j}\left(1-p_{i\to j}\,a_i^{(t)}\right).$$
Under this update, a claim stays standing to the extent that none of its standing attackers defeats it. An attack from a claim that has itself been defeated has little effect. Each candidate answer $y$ is scored by a smoothed maximum of $a_j$ over the claims $C_y$ that support it, and a softmax over these scores gives the decision. The maximum is chosen deliberately. Repeating a refuted claim many times adds nothing, because the same attacker defeats every copy, and this is exactly the property that headcount-based rules lack.
Training has two stages. First, the edge scorer is pretrained on the human argument-relation annotations in CDCP and the Stab and Gurevych essays. It is also pretrained on a silver set of debate transcripts whose attack edges were labeled by a strong LLM prompted pair by pair. Second, the model is fine-tuned end to end on debates with known correct answers, optimizing cross-entropy on the adjudicated answer plus the silver-edge loss as a regularizer. The answer-level signal is what lets the student differ from, and possibly beat, its LLM teacher. The teacher labels edges by how the text sounds. The student is rewarded for an edge graph whose surviving claims point to the true answer.
Baselines
The naive reference points are a single agent with chain-of-thought and self-consistency sampling at matched token budget, and plain majority voting over the same debate transcripts. Kaesberg et al. report 54.8% for voting on MuSR murder mysteries and 51.7% versus 54.0% for voting versus consensus on MMLU. Those numbers come from their models and protocols. We rerun both rules on our own transcripts rather than borrowing the figures, and we count voting as beaten only if graph adjudication improves on it on both task types.
The strongest existing method on math is objective memory masking (Multi-Agent Debate with Memory Masking). It reports MATH accuracy rising from 71.2% to 80.2% with Qwen2.5-Math-7B, and a 6.6-point gain on AIME. We run it with the same agents, the same number of rounds, and the same base model, ending with voting as the authors do. Graph adjudication is also applied on top of masked transcripts, to test whether the two combine. To count as a win, graph adjudication must exceed masking-plus-voting at equal or lower tokens, with confidence intervals from per-question bootstrap.
The prompt-judge baseline follows FORD (Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate), where a GPT-4 summarizing judge added about 0.6 points of accuracy. We give a strong judge model the full transcript and report its cost alongside its accuracy.
The teacher-graph baseline has the LLM annotate all attack edges at inference time and then applies the same acceptability computation. This isolates whether a trained small scorer adds anything beyond amortizing the teacher's annotation cost.
The key ablations isolate the proposal's ingredients. The flat learned selector uses the same encoder and the same answer labels, but pools claims per answer and classifies, with no graph. A symmetric scorer replaces the bilinear $\mathbf{W}$ with a symmetric matrix and removes the attacker and target priors. A pairwise cross-encoder, a natural-language-inference-style classifier reading two claims at a time, is fed into the same semantics. Finally, a silver-only scorer is trained without the end-to-end answer loss. All learned adjudicators share training debates, encoder size, and an equal hyperparameter-search budget. That last point matters because Dozat and Manning report that biaffine parsers are unusually sensitive to dropout, hidden sizes, and loss choice.
Why not just…?
Why not just train a verifier that reads the transcript and picks the answer? This is the real threat. Our flat-selector ablation is exactly this model, trained on identical labels. Ruiz-Dolz et al.'s 0.64 versus 0.33 gap between graph and flat readers is suggestive, but it comes from human debates with gold graphs, so it does not settle the question. We expect the flat selector to learn shortcuts that correlate with answer frequency and with which agent is strongest. Those shortcuts fail exactly where adjudication matters. We therefore evaluate on three targeted subsets: debates where the initial majority was wrong, transcripts where a wrong claim has been duplicated by a stubborn agent, and teams where agents swap identities. If the flat selector matches graph adjudication on these subsets too, we will report that the structure is unnecessary.
Why not just filter memory by perplexity, as MAD-M² does? Perplexity judges each utterance alone. A fluent, low-perplexity step can be logically invalid, and nothing in a single-utterance score can register that it has been refuted. The masking baseline and the masking-plus-adjudication combination show whether directed relations add information that per-utterance scoring misses. If masking alone matches us on MATH but not on MuSR, the honest conclusion is that refutation structure matters for soft reasoning more than for calculation.
Why not just ask a strong LLM which claims refute which? That is our teacher-graph baseline. It is quadratic in prompts, or it requires one very long prompt. Pitre et al. (A Diagnostic Study of Multi-Agent LLMs for Real-World Debates) and Xiong et al. document that judging with a model is vulnerable to sycophancy and to dominance by one agent. If the teacher matches the student in accuracy, our contribution narrows to a cost reduction, and we will report it as such. The cost comparison will be stated in tokens and wall-clock time.
Why not just use a pairwise cross-encoder instead of a biaffine layer? With a few dozen claims, scoring every pair is affordable, so efficiency is not our argument. Our argument is that the biaffine scorer sees every claim in the context of the whole debate, and that its attacker and target priors can encode which agents' objections tend to be reliable. The cross-encoder ablation tests whether either property matters.
Proposed Experiments
Debates are generated with five agents over three rounds. We use homogeneous teams of one open model and heterogeneous teams that mix strong and weak models. The heterogeneous teams reproduce the distraction effect Xiong et al. describe, where weaker, stubborn models derail stronger ones. Training debates come from the MATH training split and a disjoint portion of MMLU-Pro. Evaluation uses the MATH test set, held-out MMLU-Pro, and MuSR. MuSR has only 756 instances, so it is evaluated zero-shot to measure transfer. Results are broken out separately for knowledge and reasoning tasks, so the voting-versus-consensus trade-off can be checked directly.
Graph quality is measured on two fronts. We report edge F1 on about 100 debate transcripts that we will annotate by hand. We also report edge F1 on CDCP, where Morio et al. report 34.04, to confirm that decision-driven training does not destroy genuine argument parsing.
Four checks decide the outcome. First, on debates where the initial majority was wrong, Pitre et al. found that high-quality deliberation recovers the truth 74% of the time against 40% for low-quality deliberation. We measure recovery for voting, the judge, the flat selector, and graph adjudication. Second, adding duplicated wrong claims should leave graph adjudication unchanged while degrading voting and, we predict, the flat selector. Third, in heterogeneous teams, the learned target prior should mark claims from the distracting weak agent as fragile, and accuracy should approach that of the strong agent alone. Fourth, the symmetric-scorer and silver-only ablations must each lose accuracy. If they do not, the asymmetric parameterization or the end-to-end training is not doing the work, and we will say so.
References
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Automatic Debate Evaluation with Argumentation Semantics and Natural Language Argument Graph Networks
- Towards Better Non-Tree Argument Mining: Proposition-Level Biaffine Parsing with Task-Specific Parameterization
- End-to-End Argument Mining as Biaffine Dependency Parsing
- Multi-Agent Debate with Memory Masking
- Adaptive Cluster Collaborativeness Boosts LLMs Medical Decision Support Capacity
- Simpler but More Accurate Semantic Dependency Parsing
- Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
- A Diagnostic Study of Multi-Agent LLMs for Real-World Debates