RoundTable: Investigating Group Decision-Making Mechanism in Multi-Agent Collaboration

Young Min Cho, Raphael Shu, Nilaksh Das, Tamer Alkhouli, Yi-An Lai, Jason Cai, Monica Sunkara, Yi ZhangSubmission to ICLR 2025 · Rejected

TL;DR

Most multi-agent LLM systems rely on rigid, centralized hierarchies or simple majority rules to make decisions, often leading to bottlenecks and biased outcomes. RoundTable introduces a decentralized platform to evaluate how classical social choice mechanisms (e.g., ranked, rated, and plurality voting) shape agent collaboration. The authors discover that nuanced, score-based voting yields better outcomes than winner-take-all votes, and that conversational linguistic cues can accurately predict when a multi-agent discussion should stop before performance degrades.

What Problem Does This Paper Address?

When multiple LLM agents collaborate, differences in goals, specialized context, and private data inevitably cause disagreements. Most existing multi-agent systems (MAS) resolve these conflicts using centralized architectures—such as a single "manager" agent or a strict waterfall pipeline. However, centralized systems introduce significant failure modes:

  1. Bias and Misrepresentation: The deciding agent can impose its own biases and ignore valid contributions from others.
  2. Rigidity: Fixed hierarchies struggle to generalize across diverse, dynamic tasks.
  3. Incompatibility with Information Asymmetry: Central authorities cannot optimally decide when agents hold private, non-shareable data.

Decentralized group decision-making (social choice) offers a promising alternative by distributing voting power. However, it remains unclear which social choice mechanisms work best for LLMs, how voting rules affect agent communication, and how to detect when further discussion yields diminishing returns.

Methods

The authors introduce RoundTable, a turn-based collaboration platform structured into three simultaneous phases per round:

  • Message Phase: Agents openly exchange messages, share insights, and ask clarifying questions simultaneously.
  • Proposal Phase: Agents use step-by-step reasoning to submit candidate solutions (or skip).
  • Voting Phase: Agents evaluate candidates (plus the current group consensus) under a specified social choice rule.

The authors evaluate six social choice functions:

  • One-Vote Mechanisms: Unanimous, Majority, and Plurality voting.
  • Score-Based Mechanisms: Rated (5-point Likert scale), Ranked (Borda-style reciprocal points), and Cumulative voting (point-budget allocation).

To monitor collaboration dynamics, the authors track four linguistic metrics:

  • Message Length & Complexity: Measured via word count and Flesch-Kincaid readability grade.
  • Information Difference: Average cosine distance between current messages and the embedding center of the previous round (using Sentence-BERT).
  • Dialogue Acts: Classifying utterances into informational acts (Inform, Request, Confirm, Summarize, Evaluate) and decision acts (Propose, Compromise, Defend, Accept, Decline).
  • Early Stopping via Linguistic Cues: Identifying optimal stopping points using Information Difference thresholds and Ordinary Least Squares (OLS) regression over dialogue-act transition pairs ($A \to B$).

Datasets

  1. Simulated Environment (Exchange Economy): A plus-sum market game with $K$ agents and $K$ goods (100 units each) parameterized by asymmetric Cobb-Douglas utility functions. Agents negotiate good allocations to maximize individual utility without knowing the group's total utility function.
  2. Complex Environment (MovieLens-100k): 100 randomly sampled test instances from the u1.test split. Information is partitioned across three specialized agents: BasicInfo Agent (user demographics and movie metadata), UserHistory Agent (target user's historical ratings), and MovieHistory Agent (ratings from other users, ranked by collaborative matrix factorization similarity).

Key Insights and Ideas

  • Score-Based vs. One-Vote Mechanisms: Score-based mechanisms (Rated, Ranked, Cumulative) consistently beat one-vote rules in early rounds. Allowing agents to express gradient preferences creates dynamic updates and smoother compromise, whereas one-vote schemes cause deadlock and high disagreement.
  • The Pitfalls of Extreme Strictness: Unanimous voting achieves the highest fairness when an agreement is reached, but it is exceedingly rigid (93% rigidity), resulting in sluggish consensus and poor overall utility. Moderate decision flexibility (e.g., Majority or Rated) achieves the best trade-off between speed and solution quality.
  • The "V-Shaped" Over-Collaboration Curve: Agent performance frequently improves in early rounds but stagnates or degrades in later rounds (showing an inverted-U in utility, or a V-shape in prediction error). Prolonged interactions introduce cognitive clutter and topic drift.
  • Language as a Stopping Signal: As conversations mature, novel information decays steadily (Information Difference drops) and dialogue shifts from exploratory loops (Request $\leftrightarrow$ Propose) to closure (Accept $\to$ End). Using these linguistic signals to halt conversation avoids degradation without running a fixed 10-round loop.

Key Results

  • Exchange Economy:
  • Score-based mechanisms reached high utility much faster: by Round 3, Ranked voting achieved a group total utility of 80.92, and Rated achieved 80.83, whereas Unanimous voting lagged at 33.94.
  • Final group utility peaked in middle rounds (e.g., Majority utility was 81.33 at Round 5 vs. 79.61 at Round 10).
  • MovieLens Recommendation:
  • Decentralized LLM collaboration performed comparably to a supervised collaborative filtering baseline with temporal features.
  • One-vote mechanisms plateaued early (Majority MAE: 0.81 at Round 3 $\to$ 0.86 at Round 10), while Rated voting maintained continuous refinement, achieving an MAE of 0.75 at Round 6.
  • Early Stopping:
  • Stopping based on linguistic metrics outperformed the full 10-round baseline (@10 MAE = 0.86 for Majority). Halting via Information Difference achieved an MAE of 0.78 (near the theoretical Oracle of 0.63).

Limitations and Future Work

  • Agent Scalability Bottlenecks: Performance dropped as agent group size increased ($K = 3 \to 5$). Majority voting suffered disproportionately with even numbers of agents ($K=4$) due to tie-breaking difficulties.
  • Limited Prompt Diversity: Agent profiles and utility definitions were manually assigned rather than dynamically recruited or adapted during run-time.
  • Hybrid Mechanisms: The authors evaluated static voting protocols throughout each run; exploring adaptive, staged social choice mechanisms (e.g., score-based exploration transitioning to unanimous validation) remains an open avenue for future work.

Summary updated

Abstract

This study investigates the efficacy of Multi-Agent Systems in eliciting cross-agent communication and enhancing collective intelligence through group decision-making in a decentralized setting. Unlike centralized mechanisms, where a fixed hierarchy governs social choice, decentralized group decision-making allows agents to engage in joint deliberation. Our research focuses on the dynamics of communication and decision-making within various social choice methods. By applying different voting rules in various environments, we find that moderate decision flexibility yields better outcomes. Additionally, exploring the linguistic features of agent-to-agent conversations reveals indicators of effective collaboration, offering insights into communication patterns that facilitate or hinder collaboration. Finally, we propose various methods for determining the optimal stopping point in multi-agent collaborations based on linguistic cues. Our findings contribute to a deeper understanding of how decentralized decision-making and group conversation shape multi-agent collaboration, with implications for the design of more effective MAS environments.
BibTeX
@misc{cho2024roundtable,
  author = {Young Min Cho and Raphael Shu and Nilaksh Das and Tamer Alkhouli and Yi-An Lai and Jason Cai and MONICA SUNKARA and Yi Zhang},
  title = {{RoundTable: Investigating Group Decision-Making Mechanism in Multi-Agent Collaboration}},
  year = {2024},
  eprint = {2411.07161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.07161},
}
Figure