Abstract
Recent advancements in autonomous multi-agent systems (MAS) based on large language models (LLMs) have enhanced the application scenarios and improved the capability of LLMs to handle complex tasks. Despite demonstrating effectiveness, existing studies still evidently struggle to evaluate, analysis, and reproducibility of LLM-based MAS. In this paper, to facilitate the research on LLM-based MAS, we introduce an open, scalable, and real-time updated platform for accessing and analyzing the LLM-based MAS based on the games "Who is Spy?" (WiS). Our platform is featured with three main worths: (1) a unified model evaluate interface that supports models available on Hugging Face; (2) real-time updated leaderboard for model evaluation; (3) a comprehensive evaluation covering game-winning rates, attacking, defense strategies, and reasoning of LLMs. To rigorously test WiS, we conduct extensive experiments coverage of various open- and closed-source LLMs, we find that different agents exhibit distinct and intriguing behaviors in the game. The experimental results demonstrate the effectiveness and efficiency of our platform in evaluating LLM-based MAS. Our platform and its documentation are publicly available at https://whoisspy.ai/
Introduction
The rapid advancement of Large Language Models (LLMs) has paved the way for autonomous multi-agent systems (MAS) capable of handling complex, collaborative tasks. However, evaluating these systems remains a significant challenge. Traditional benchmarks often rely on static datasets or narrow tool-use scenarios that fail to capture the nuances of social interaction, "theory of mind," and strategic deception. To bridge this gap, we introduce the WiS (Who is Spy?) platform—an open, scalable, and real-time updated environment designed to analyze the reasoning, deception, and adversarial capabilities of LLM agents within a dynamic social deduction game.
The WiS Platform Architecture
The WiS platform provides a unified interface for model evaluation, supporting both open-source models from Hugging Face and various closed-source APIs. The system is composed of several key components: a web-based interface for community interaction and competition, a Python SDK for seamless agent integration, and a real-time leaderboard.
The platform facilitates large-scale experiments by providing stable environments for fair comparison. It moves beyond simple win/loss metrics by offering deep dives into attacking strategies, defense mechanisms, and the underlying reasoning processes of the agents.
Evaluation Methodology
The core of the evaluation is the social deduction game "Who is Spy?". In a standard round, six players participate: five Civilians and one Spy. Civilians receive a target word, while the Spy receives a related but different word. Players must describe their word without revealing it directly, then vote to eliminate the suspected Spy.
To rigorously test agent capabilities, the platform employs three specialized evaluation strategies:
- Prompt Injection Attacking (PIA): Spies attempt to insert malicious instructions into their descriptions to trick other models into committing game "fouls," such as repeating descriptions.
- Prompt Injection Defense (PID): Agents are evaluated on their ability to resist such manipulation and ignore deceptive instructions.
- Reasoning Analysis: Models are required to output their internal reasoning before voting, allowing for an assessment of their logical consistency and "theory of mind."
Experimental Results and Insights
Our empirical study involved 200 participants and 204 distinct AI agents across nearly 16,000 game rounds. We benchmarked 10 prominent models, ranging from state-of-the-art closed-source models like GPT-4o to powerful open-source alternatives like Qwen2.5-72B-Instruct.
Key Findings
- The Deception Gap: Across all models, win rates for the Spy role were significantly lower than for Civilians. This suggests that while current LLMs excel at logical tasks, they still struggle with the complex social manipulation required to "blend in" effectively.
- Top Performers: GPT-4o dominated the leaderboard with an overall win rate of 76.67%. Among open-source models, Qwen2.5-72B-Instruct emerged as a champion, achieving a 70.00% win rate and rivaling the performance of GPT-4.
- Reasoning as a Double-Edged Sword: While Chain-of-Thought reasoning improved voting accuracy for top-tier models like GPT-4o, it often hindered mid-tier models. Flawed logic in these models frequently confused their civilian teammates, leading to poorer overall performance.
- Adversarial Vulnerability: Even the most advanced models showed fragility. Under targeted "Prompt Injection Defense" attacks, GPT-4o’s win rate dropped dramatically, highlighting a critical need for improved robustness in multi-agent adversarial settings.
Conclusion
The WiS platform offers a comprehensive and scalable framework for evaluating the social and strategic dimensions of LLM-based multi-agent systems. Our findings reveal that while LLMs are becoming increasingly capable, significant hurdles remain in areas of deception, social reasoning, and adversarial robustness. Future work will focus on refining these metrics and exploring even more complex social deduction scenarios to better understand and improve strategic AI behavior.
Summary updated