TL;DR
Most multi-agent LLM systems rely on rigid, centralized hierarchies or simple majority rules to make decisions, often leading to bottlenecks and biased outcomes. RoundTable introduces a decentralized platform to evaluate how classical social choice mechanisms (e.g., ranked, rated, and plurality voting) shape agent collaboration. The authors discover that nuanced, score-based voting yields better outcomes than winner-take-all votes, and that conversational linguistic cues can accurately predict when a multi-agent discussion should stop before performance degrades.
What Problem Does This Paper Address?
When multiple LLM agents collaborate, differences in goals, specialized context, and private data inevitably cause disagreements. Most existing multi-agent systems (MAS) resolve these conflicts using centralized architectures—such as a single "manager" agent or a strict waterfall pipeline. However, centralized systems introduce significant failure modes:
- Bias and Misrepresentation: The deciding agent can impose its own biases and ignore valid contributions from others.
- Rigidity: Fixed hierarchies struggle to generalize across diverse, dynamic tasks.
- Incompatibility with Information Asymmetry: Central authorities cannot optimally decide when agents hold private, non-shareable data.
Decentralized group decision-making (social choice) offers a promising alternative by distributing voting power. However, it remains unclear which social choice mechanisms work best for LLMs, how voting rules affect agent communication, and how to detect when further discussion yields diminishing returns.
Methods
The authors introduce RoundTable, a turn-based collaboration platform structured into three simultaneous phases per round:
- Message Phase: Agents openly exchange messages, share insights, and ask clarifying questions simultaneously.
- Proposal Phase: Agents use step-by-step reasoning to submit candidate solutions (or skip).
- Voting Phase: Agents evaluate candidates (plus the current group consensus) under a specified social choice rule.
The authors evaluate six social choice functions:
- One-Vote Mechanisms: Unanimous, Majority, and Plurality voting.
- Score-Based Mechanisms: Rated (5-point Likert scale), Ranked (Borda-style reciprocal points), and Cumulative voting (point-budget allocation).
To monitor collaboration dynamics, the authors track four linguistic metrics:
- Message Length & Complexity: Measured via word count and Flesch-Kincaid readability grade.
- Information Difference: Average cosine distance between current messages and the embedding center of the previous round (using Sentence-BERT).
- Dialogue Acts: Classifying utterances into informational acts (Inform, Request, Confirm, Summarize, Evaluate) and decision acts (Propose, Compromise, Defend, Accept, Decline).
- Early Stopping via Linguistic Cues: Identifying optimal stopping points using Information Difference thresholds and Ordinary Least Squares (OLS) regression over dialogue-act transition pairs ($A \to B$).
Datasets
- Simulated Environment (Exchange Economy): A plus-sum market game with $K$ agents and $K$ goods (100 units each) parameterized by asymmetric Cobb-Douglas utility functions. Agents negotiate good allocations to maximize individual utility without knowing the group's total utility function.
- Complex Environment (MovieLens-100k): 100 randomly sampled test instances from the
u1.testsplit. Information is partitioned across three specialized agents:BasicInfo Agent(user demographics and movie metadata),UserHistory Agent(target user's historical ratings), andMovieHistory Agent(ratings from other users, ranked by collaborative matrix factorization similarity).
Key Insights and Ideas
- Score-Based vs. One-Vote Mechanisms: Score-based mechanisms (Rated, Ranked, Cumulative) consistently beat one-vote rules in early rounds. Allowing agents to express gradient preferences creates dynamic updates and smoother compromise, whereas one-vote schemes cause deadlock and high disagreement.
- The Pitfalls of Extreme Strictness: Unanimous voting achieves the highest fairness when an agreement is reached, but it is exceedingly rigid (93% rigidity), resulting in sluggish consensus and poor overall utility. Moderate decision flexibility (e.g., Majority or Rated) achieves the best trade-off between speed and solution quality.
- The "V-Shaped" Over-Collaboration Curve: Agent performance frequently improves in early rounds but stagnates or degrades in later rounds (showing an inverted-U in utility, or a V-shape in prediction error). Prolonged interactions introduce cognitive clutter and topic drift.
- Language as a Stopping Signal: As conversations mature, novel information decays steadily (Information Difference drops) and dialogue shifts from exploratory loops (Request $\leftrightarrow$ Propose) to closure (Accept $\to$ End). Using these linguistic signals to halt conversation avoids degradation without running a fixed 10-round loop.
Key Results
- Exchange Economy:
- Score-based mechanisms reached high utility much faster: by Round 3, Ranked voting achieved a group total utility of 80.92, and Rated achieved 80.83, whereas Unanimous voting lagged at 33.94.
- Final group utility peaked in middle rounds (e.g., Majority utility was 81.33 at Round 5 vs. 79.61 at Round 10).
- MovieLens Recommendation:
- Decentralized LLM collaboration performed comparably to a supervised collaborative filtering baseline with temporal features.
- One-vote mechanisms plateaued early (Majority MAE: 0.81 at Round 3 $\to$ 0.86 at Round 10), while Rated voting maintained continuous refinement, achieving an MAE of 0.75 at Round 6.
- Early Stopping:
- Stopping based on linguistic metrics outperformed the full 10-round baseline (@10 MAE = 0.86 for Majority). Halting via Information Difference achieved an MAE of 0.78 (near the theoretical Oracle of 0.63).
Limitations and Future Work
- Agent Scalability Bottlenecks: Performance dropped as agent group size increased ($K = 3 \to 5$). Majority voting suffered disproportionately with even numbers of agents ($K=4$) due to tie-breaking difficulties.
- Limited Prompt Diversity: Agent profiles and utility definitions were manually assigned rather than dynamically recruited or adapted during run-time.
- Hybrid Mechanisms: The authors evaluated static voting protocols throughout each run; exploring adaptive, staged social choice mechanisms (e.g., score-based exploration transitioning to unanimous validation) remains an open avenue for future work.
Summary updated