Phantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-Making

Disha Sheshanarayana, Rajat Subhra Pal, Manjira Sinha, Tirthankar DasguptaarXiv 2026 · arXiv

TL;DR

Multi-agent LLM systems can fall victim to "phantom consensus"—a failure mode where agents publicly converge on an incorrect decision they privately rejected, driven by misperceived group beliefs. This paper introduces PHANTOM-BENCH to measure this social reasoning flaw and demonstrates that while ambiguous social signals can slash collective accuracy by nearly 50%, revealing agents' true private judgments can effectively repair the group's decision-making.

What Problem Does This Paper Address?

The promise of multi-agent LLM systems—where multiple models deliberate to reach a final answer—rests on the "wisdom of the crowd": independent agents should correct each other's errors. However, in human groups, social context often undermines independence through phenomena like pluralistic ignorance, where individuals publicly comply with a perceived norm they privately disagree with (the "Abilene Paradox").

The authors investigate whether LLMs suffer from a similar "phantom consensus." They ask: when agents observe ambiguous public comments from peers, do they form false beliefs about what others privately think? And can these misperceptions drive a collective failure toward a "phantom target" that no agent actually intended to support?

Methods

To isolate this effect, the researchers developed a structured evaluation framework that separates an agent's internal judgment from its public behavior.

Figure 1: Belief-shift mechanism in PHANTOM-BENCH. Agents first make independent private decisions, then observe controlled ambiguous public signals and report both a public decision and a perceived group-private belief. The setup tests whether agents retain their private judgment or shift toward an inferred group belief unsupported by the group’s recorded private decisions.

The core of their approach is a five-condition experimental design:

  1. Private: Agents decide in total isolation (the baseline).
  2. Observed Public: Agents see ambiguous peer comments and must report both their public decision and their estimate of what the group privately believes.
  3. Informed Transparency: Agents are shown the actual private decisions of their peers before making a final choice.
  4. Neutral Public (Control): Agents see "filler" comments to ensure failures aren't just caused by seeing more text.
  5. Explicit Public Vote (Control): Agents see clear majority recommendations to distinguish "phantom" belief inference from simple vote-following.
Figure 2: Overview of the five belief-context conditions in PHANTOM-BENCH. All agents receive the same scenario and decision request, but the group context varies across conditions: no group information, ambiguous public signals, recorded private decisions revealed under transparency, neu- tral public comments without an implied norm, or explicit peer recommendations. The setup sepa- rates private judgments, public decisions, perceived group-private beliefs, and recovery under trans- parency.

Datasets

The authors introduced PHANTOM-BENCH, a benchmark consisting of 480 scenarios (yielding 1,920 agent-level examples per condition). These scenarios span five critical domains:

  • Financial compliance
  • Content moderation
  • Hiring fairness
  • Healthcare privacy
  • Workplace safety

Each scenario includes a "gold" private decision and a "phantom target"—a plausible but incorrect alternative that ambiguous social signals are designed to suggest.

Key Insights and Ideas

  • Social Fragility is Real: LLMs aren't just "bad" at these tasks; they are socially suggestible. Even when a model is perfectly capable of solving a problem privately, it will often abandon that correct answer if it thinks its peers disagree.
  • Second-Order Belief Errors: The failure is rooted in "theory of mind" errors. Models consistently misinterpret ambiguous comments as evidence of a consensus that doesn't exist.
  • Transparency as a Cure: Simply revealing what agents actually thought in private (without revealing the "correct" answer) fixes most failures. This suggests that preserving "first-pass" independent judgments is crucial for robust multi-agent system design.
  • Model-Dependent Susceptibility: Open-weight models (like Qwen-2.5) were significantly more prone to phantom consensus than closed-weight models (like Gemini), suggesting that alignment or RLHF processes might influence social resilience.

Key Results

The impact of ambiguous social signals was dramatic across all nine tested models:

  • Accuracy Collapse: In the most extreme case (Qwen-2.5-7B), accuracy plummeted from 77.0% (private) to just 29.2% (public)—a 47.8 percentage point drop.
  • Collective Rationality Failure: The Collective Rationality Failure Rate (CRFR)—scenarios where at least 3 of 4 agents were privately correct but at most 1 remained correct publicly—reached 47.3% for the most susceptible models.
  • Phantom Alignment: Many models didn't just get the answer wrong; they specifically moved toward the "phantom target." For Qwen-2.5-7B, nearly 40% of all scenarios resulted in a collective shift specifically to the incorrect phantom target.
Quantitative metrics showing the gap between private and public performance across various models.

Limitations and Future Work

The authors acknowledge several constraints:

  • Action Space: The benchmark uses a simplified three-way choice (Proceed, Block, Defer), whereas real-world decisions are often more nuanced.
  • Static Interaction: The study uses a fixed four-agent setup with one-shot public signals rather than a dynamic, multi-round "live" deliberation.
  • Human-like Reasoning: The authors clarify they are not claiming LLMs have human-like "theory of mind," but rather that they exhibit behavioral patterns that look like pluralistic ignorance.

Future work could explore richer decision spaces, different group hierarchies (e.g., "senior" vs "junior" agents), and whether specific prompting techniques can "inoculate" agents against social pressure.

Summary updated

Abstract

Multi-agent LLM systems increasingly aggregate judgments across model instances, but reasoning about other agents can itself introduce failure. We study a setting in which LLMs shift their public decisions after observing ambiguous social signals while reporting inaccurate estimates of what other agents privately judged, even when the group's recorded private judgments do not support the apparent consensus. We call this failure phantom consensus and study it as a pluralistic-ignorance-style pattern in controlled LLM collectives. To measure this phenomenon, we introduce PHANTOM-Bench, a benchmark that separates isolated private judgments, public decisions after ambiguous social signals, elicited estimates of the group's private judgment, transparency-based recovery, and controls for neutral public text and explicit vote-following. Across nine closed- and open-weight models, ambiguous public signals consistently degrade decision quality and can turn correct private group judgments into incorrect public collective outcomes. Public accuracy drops by as much as 47.8 percentage points, and collective rationality failures reach 47.3% in the most susceptible setting. We observe substantial errors in reported group-private beliefs alongside public-stage failures, while control conditions reveal model-dependent differences in susceptibility to ambiguous and explicit peer signals. Revealing the agents' recorded private decisions repairs many failures without revealing the gold label, although recovery varies across models. Our results show that evaluating multi-agent LLM systems requires measuring not only final agreement, but whether that agreement reflects independently measured private judgments or an apparent consensus.
BibTeX
@misc{sheshanarayana2026phantom,
  author = {Disha Sheshanarayana and Rajat Subhra Pal and Manjira Sinha and Tirthankar Dasgupta},
  title = {{Phantom Consensus: Measuring Social Reasoning in Multi-Agent LLM Decision-Making}},
  year = {2026},
  eprint = {2611.08971},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08971},
}
Figure