TL;DR
Multi-agent LLM debate doesn't magically create "collective intelligence" beyond its members; instead, it functions primarily through capability transfer, where a stronger model "teaches" weaker ones. This gain is strictly gated by the capability gap between models and the receiver's willingness to listen, meaning a panel rarely outperforms its strongest member and remains highly vulnerable to influential but incorrect peers.
What Problem Does This Paper Address?
In the quest to squeeze more performance out of Large Language Models (LLMs), researchers often use "multi-agent debate" (MAD). The standard narrative is that by putting different models in a room to argue, their diverse perspectives will compose into a "collective reasoning" capability that exceeds what any single model could do alone.
However, recent evaluations have shown that debate often loses to simple single-agent inference or smart routing, often at a fraction of the cost. This paper tackles the fundamental question: Does heterogeneous debate actually create new capability through diversity, or is it just a fancy way of relaying the answer from the smartest model in the room?
Methods
The authors move away from vague notions of "diversity" and propose a rigorous framework based on Capability Transfer. They test four concrete predictions using matched three-agent panels (two "receivers" and one "peer").
They compare four conditions:
- Receiver-Homogeneous: All three agents are the same (the control).
- Peer-Homogeneous: All three agents are the stronger model (the reference).
- Honest Heterogeneous: Two weaker receivers and one stronger "honest" peer.
- Adversarial Heterogeneous: The same peer, but instructed to defend a plausible wrong answer.
The researchers define two key variables:
- Capability Gap ($\Gamma_d$): The accuracy difference between the peer and the receiver.
- Reception ($\rho_d$): The probability that a wrong receiver becomes correct when the peer provides a correct signal.
They then model the gain in accuracy ($\Delta A$) using a simple decomposition: $$\Delta A \approx \kappa \cdot \Gamma_d^+ \cdot \rho_d$$ Essentially, the gain is the product of how much better the peer is and how much the receiver is willing to learn from them.
Datasets
The study evaluates these panels across several challenging reasoning benchmarks:
- MATH-hard: Levels 4 and 5 of the MATH-500 dataset.
- GSM8K: Grade school math word problems.
- MuSiQue-Ans: A multi-hop question-answering dataset used as a "scope probe" to test regimes where the peer is actually weaker than the receivers (negative gap).
The models used range from small open-weights (Qwen-7B, Llama-3.2-1B) to frontier APIs (GPT-4.1, Gemini-2.5-Flash) and massive open models (GPT-OSS-120B).
Key Insights and Ideas
- The Capability Gate: Diversity alone provides almost zero benefit. If the added peer isn't significantly better than the existing models ($\Gamma \le 0$), the net gain is negligible, even if the models are "diverse."
- Reception is the Conversion Rate: A model can be a "genius" peer, but if the receiver is stubborn (low reception), the capability isn't transferred. Interestingly, reception remains relatively stable for a given receiver even as the peer gets stronger.
- The "Borrowed Ceiling": In all tested settings, the panel's majority-vote accuracy never exceeded the best member's individual accuracy. The "oracle" (the best possible answer if you could pick any member's response) was much higher, but the debate process failed to realize that potential.
- Imported Risk: The same channel that delivers "borrowed strength" also imports risk. If an influential peer is assigned an adversarial role, it doesn't just fail to help; it can actively corrupt the panel, pushing it below the performance of a single-model baseline.
Key Results
- Predictive Accuracy: The authors' simple "Gap $\times$ Reception" formula achieved an $R^2$ of 0.995, suggesting that capability transfer almost entirely explains the gains seen in heterogeneous debate.
- The MuSiQue Probe: When Qwen-7B (weaker) was the peer for Gemma-3-12B (stronger) on MuSiQue, the gain was a measly +1.17 points, despite a high reception rate of 62.5%. This confirms that without a positive capability gap, reception is useless.
- Adversarial Impact: Reversing a peer's role from honest to adversarial caused a massive swing. In one case (GPT-OSS peer), the panel went from +21.37 points above the baseline to 6.23 points below it.
- Efficiency: Five-round debate was found to be incredibly expensive, using 40 to 86 times more tokens than a single call to the strongest model, while rarely beating that model's standalone performance.
Limitations and Future Work
- Finite Directions: The study focused on nine primary directions. While the results are robust, they don't cover every possible model pair or task.
- Structural Coupling: The authors acknowledge that the Capability Gap and Reception aren't perfectly independent variables, which adds some structural complexity to their formula.
- Simple Adversaries: The adversarial model used a "committed-wrong" prompt. The study does not explore more sophisticated, coordinated, or adaptive multi-peer attacks.
- Future Directions: The authors suggest that instead of "blind" debate, future work should focus on learned routing or selection-oriented systems that can identify which model has the correct answer rather than trying to force a consensus through conversation.
Summary updated