TL;DR
Evaluating large language models (LLMs) via static benchmarks often suffers from data contamination and fails to capture interactive, human-like social intelligence. To address this, the authors introduce SpyGPT and SpyGame, evaluation frameworks modeled after the social deduction word-guessing game "Who is Spy" (similar to Spyfall). By testing how well models balance clear expression (aggressive mode) with strategic deception (conservative disguise mode) and multi-agent deduction, this framework effectively differentiates model intelligence and Theory of Mind capabilities.
What Problem Does This Paper Address?
Traditional LLM evaluations—such as MMLU, GLUE, or human-annotated instruction datasets like AlpacaEval—are costly, hard to scale dynamically, and prone to benchmark leakage (contamination). More importantly, static NLP tasks fail to measure communicative intelligence: the ability to infer other agents' intentions, engage in strategic deception, adapt dynamically to conversation history, and demonstrate Theory of Mind (ToM).
Methods
The authors propose two complementary frameworks:
- SpyGPT (Single-Agent Expression & Disguise):
- Aggressive Mode: The target LLM is prompted to provide a precise, 100-word description of a keyword so a listener can easily guess it.
- Conservative Mode: Using Chain-of-Thought (CoT), the LLM infers similar candidate words and generates a terse, ambiguous description (≤10 words) focused strictly on high-level commonalities to disguise the true word.
- Judging: GPT-4 rates (1–5) how accurately descriptions match the target word versus pre-defined distractor words.
- SpyGame (Interactive Multi-Agent Deduction):
- Built on the party game "Who is Spy", a 4-player game with 3 villagers and 1 spy. The villagers receive one keyword (e.g., BERT), while the spy receives a semantically close sibling word (e.g., GPT). The guest agent under evaluation is assigned the spy role without being told it is the spy.
- Agent Actions: In each round, agents perform Word Guessing (inferring rivals' secret words), Speaking (describing their own word), Reasoning (deducing identities), and Voting (casting an elimination vote).
- Bias Mitigation: LLM agents exhibit systematic voting biases. The authors identify and control for three key biases:
- Name Bias: Models favor voting for certain human names; mitigated using neutral IDs (
Player 1toPlayer 4). - Speaking Order Bias: Early speakers receive more votes (Position 1: 32.5% vs. Position 4: 19.6%); mitigated by randomizing speaking order.
- Option Order Bias: LLMs heavily favor the first option presented in a choice list (Position 1: 43.4% vs. Position 2: 6.6%); mitigated by shuffling options.
[Keyword Pair: BERT vs. GPT]
│
┌─────────────┴─────────────┐
▼ ▼
Villagers (GPT-3.5) Spy (Target LLM)
│ │
└───────────► Round ◄───────┘
│
1. Word Guessing (Infer rival words)
2. Speaking (Accurate vs. Ambiguous)
3. Reasoning (Theory-of-Mind deduction)
4. Voting (Identify and eliminate spy)
Datasets
The authors collected two specialized datasets across English and Chinese spanning social and scientific domains:
- SpyGPT Corpus: 40 target words paired with distractors to test description and disguise capabilities.
- SpyGame Keyword Pairs: 50 semantically adjacent word pairs ($50 \times 2 = 100$ words total, e.g., BERT / GPT) where subtle semantic distinctions must be hidden or exposed.
Key Insights and Ideas
- Intelligence requires strategic ambiguity: True communicative ability is not just generating maximum clarity; advanced intelligence requires knowing when and how to be vague to conceal private information while remaining plausible.
- Severe multi-agent evaluation biases: Without strictly randomizing speaking order and voting prompt layouts, multi-agent LLM benchmarks reflect artifactual positional biases rather than genuine deduction.
- Theory of Mind emergence correlates with model scale: Spy success relies on first-order ToM ("What is Player 1's word?") and second-order ToM ("What do others think my word is?"). SOTA models actively simulate opponents' mental states to adjust their speaking strategies.
Key Results
- SpyGPT Evaluation:
- In Aggressive Mode, GPT-4 achieved a perfect 5.00 target accuracy and a low 1.22 distractor score, outperforming open-source models (e.g., Baichuan-7B at 4.08 / 1.35; Llama-2-7B-chat at 4.78 / 1.29).
- In Conservative Mode, GPT-4 successfully generated ambiguous disguises, scoring lowest on direct target reveal (4.38) and highest on distractor overlap (3.06). Human evaluation confirmed the validity of GPT-4's ratings.
- SpyGame Benchmark:
- When evaluated as the spy against three GPT-3.5-Turbo villagers, GPT-4 significantly outperformed all baseline models, achieving a 33% win rate (surviving an average of 2.18 rounds with 1.31 votes/round), compared to GPT-3.5-Turbo (21% win rate, 2.04 rounds) and Text-Davinci-002 (16% win rate, 1.99 rounds).
- Ablation Studies:
- Disabling the Word Guessing action dropped GPT-4's win rate from 0.33 to 0.26.
- Disabling the Reasoning action dropped GPT-4's win rate to 0.21, proving that active inference is essential to deceptive survival.
- Theory of Mind Metrics:
- GPT-4 dominated in 1st-order word deduction (0.38 vs. 0.17 for GPT-3.5) and 2nd-order identity deduction (0.67 vs. 0.47 for Text-Davinci-002).
Limitations and Future Work
- Host Model Asymmetry: In SpyGame, the host villagers were exclusively GPT-3.5-Turbo. Testing interactions with heterogeneous hosts or open-source populations was not explored.
- Dataset Scale: The benchmark relies on a modest set of 50 word pairs (100 words) and 40 evaluation terms; larger, domain-specialized vocabularies are needed.
- Missing Open-Source Agent Evaluations in SpyGame: While open-source models (Llama-2, Vicuna, ChatGLM2) were tested in the single-agent SpyGPT setup, full multi-agent interactive SpyGame experiments were only reported for closed-source OpenAI models, likely due to long-context instruction-following hurdles in multi-agent game loops.
Summary updated