Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models

Tian Liang, Zhiwei He, Jen-tse Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, Xing WangSubmission to ICLR 2024 · Withdrawn

TL;DR

Evaluating large language models (LLMs) via static benchmarks often suffers from data contamination and fails to capture interactive, human-like social intelligence. To address this, the authors introduce SpyGPT and SpyGame, evaluation frameworks modeled after the social deduction word-guessing game "Who is Spy" (similar to Spyfall). By testing how well models balance clear expression (aggressive mode) with strategic deception (conservative disguise mode) and multi-agent deduction, this framework effectively differentiates model intelligence and Theory of Mind capabilities.

What Problem Does This Paper Address?

Traditional LLM evaluations—such as MMLU, GLUE, or human-annotated instruction datasets like AlpacaEval—are costly, hard to scale dynamically, and prone to benchmark leakage (contamination). More importantly, static NLP tasks fail to measure communicative intelligence: the ability to infer other agents' intentions, engage in strategic deception, adapt dynamically to conversation history, and demonstrate Theory of Mind (ToM).

Methods

The authors propose two complementary frameworks:

  1. SpyGPT (Single-Agent Expression & Disguise):
  2. Aggressive Mode: The target LLM is prompted to provide a precise, 100-word description of a keyword so a listener can easily guess it.
  3. Conservative Mode: Using Chain-of-Thought (CoT), the LLM infers similar candidate words and generates a terse, ambiguous description (≤10 words) focused strictly on high-level commonalities to disguise the true word.
  4. Judging: GPT-4 rates (1–5) how accurately descriptions match the target word versus pre-defined distractor words.
  5. SpyGame (Interactive Multi-Agent Deduction):
  6. Built on the party game "Who is Spy", a 4-player game with 3 villagers and 1 spy. The villagers receive one keyword (e.g., BERT), while the spy receives a semantically close sibling word (e.g., GPT). The guest agent under evaluation is assigned the spy role without being told it is the spy.
  7. Agent Actions: In each round, agents perform Word Guessing (inferring rivals' secret words), Speaking (describing their own word), Reasoning (deducing identities), and Voting (casting an elimination vote).
  8. Bias Mitigation: LLM agents exhibit systematic voting biases. The authors identify and control for three key biases:
  9. Name Bias: Models favor voting for certain human names; mitigated using neutral IDs (Player 1 to Player 4).
  10. Speaking Order Bias: Early speakers receive more votes (Position 1: 32.5% vs. Position 4: 19.6%); mitigated by randomizing speaking order.
  11. Option Order Bias: LLMs heavily favor the first option presented in a choice list (Position 1: 43.4% vs. Position 2: 6.6%); mitigated by shuffling options.
       [Keyword Pair: BERT vs. GPT]
                 │
   ┌─────────────┴─────────────┐
   ▼                           ▼
Villagers (GPT-3.5)       Spy (Target LLM)
   │                           │
   └───────────► Round ◄───────┘
                 │
    1. Word Guessing (Infer rival words)
    2. Speaking (Accurate vs. Ambiguous)
    3. Reasoning (Theory-of-Mind deduction)
    4. Voting (Identify and eliminate spy)

Datasets

The authors collected two specialized datasets across English and Chinese spanning social and scientific domains:

  • SpyGPT Corpus: 40 target words paired with distractors to test description and disguise capabilities.
  • SpyGame Keyword Pairs: 50 semantically adjacent word pairs ($50 \times 2 = 100$ words total, e.g., BERT / GPT) where subtle semantic distinctions must be hidden or exposed.

Key Insights and Ideas

  • Intelligence requires strategic ambiguity: True communicative ability is not just generating maximum clarity; advanced intelligence requires knowing when and how to be vague to conceal private information while remaining plausible.
  • Severe multi-agent evaluation biases: Without strictly randomizing speaking order and voting prompt layouts, multi-agent LLM benchmarks reflect artifactual positional biases rather than genuine deduction.
  • Theory of Mind emergence correlates with model scale: Spy success relies on first-order ToM ("What is Player 1's word?") and second-order ToM ("What do others think my word is?"). SOTA models actively simulate opponents' mental states to adjust their speaking strategies.

Key Results

  • SpyGPT Evaluation:
  • In Aggressive Mode, GPT-4 achieved a perfect 5.00 target accuracy and a low 1.22 distractor score, outperforming open-source models (e.g., Baichuan-7B at 4.08 / 1.35; Llama-2-7B-chat at 4.78 / 1.29).
  • In Conservative Mode, GPT-4 successfully generated ambiguous disguises, scoring lowest on direct target reveal (4.38) and highest on distractor overlap (3.06). Human evaluation confirmed the validity of GPT-4's ratings.
  • SpyGame Benchmark:
  • When evaluated as the spy against three GPT-3.5-Turbo villagers, GPT-4 significantly outperformed all baseline models, achieving a 33% win rate (surviving an average of 2.18 rounds with 1.31 votes/round), compared to GPT-3.5-Turbo (21% win rate, 2.04 rounds) and Text-Davinci-002 (16% win rate, 1.99 rounds).
  • Ablation Studies:
  • Disabling the Word Guessing action dropped GPT-4's win rate from 0.33 to 0.26.
  • Disabling the Reasoning action dropped GPT-4's win rate to 0.21, proving that active inference is essential to deceptive survival.
  • Theory of Mind Metrics:
  • GPT-4 dominated in 1st-order word deduction (0.38 vs. 0.17 for GPT-3.5) and 2nd-order identity deduction (0.67 vs. 0.47 for Text-Davinci-002).

Limitations and Future Work

  • Host Model Asymmetry: In SpyGame, the host villagers were exclusively GPT-3.5-Turbo. Testing interactions with heterogeneous hosts or open-source populations was not explored.
  • Dataset Scale: The benchmark relies on a modest set of 50 word pairs (100 words) and 40 evaluation terms; larger, domain-specialized vocabularies are needed.
  • Missing Open-Source Agent Evaluations in SpyGame: While open-source models (Llama-2, Vicuna, ChatGLM2) were tested in the single-agent SpyGPT setup, full multi-agent interactive SpyGame experiments were only reported for closed-source OpenAI models, likely due to long-context instruction-following hurdles in multi-agent game loops.

Summary updated

Abstract

The automatic evaluation of LLM-based agent intelligence is critical in developing advanced LLM-based agents. Although considerable effort has been devoted to developing human-annotated evaluation datasets, such as AlpacaEval, existing techniques are costly, time-consuming, have limited scalability, and lack adaptability. In this paper, inspired by the popular language games ``Who is Spy'' and ``SpyFall'', we propose to use the word guessing game to assess the intelligence performance of LLMs. Given a word, the LLM is asked to describe the word and determine its identity (spy or not) based on its and other players' descriptions. Ideally, an advanced agent should possess the ability to accurately describe a given word using an aggressive description while concurrently maximizing confusion in the conservative description, enhancing its participation in the game. To this end, we first develop SpyGPT to evaluate LLMs' expression and disguising abilities. SpyGPT requires the target LLM to describe the given word in aggressive and conservative modes and utilizes the SOTA GPT-4 to determine whether the descriptive sentences can accurately describe the given word. We then introduce SpyGame, an interactive multi-agent framework designed to assess LLMs intelligence through participation in a competitive language-based board game. Incorporating multi-agent interaction, SpyGame requires the target LLM to possess linguistic skills and strategic thinking, providing a more comprehensive evaluation of LLMs' human-like cognitive abilities and adaptability in complex communication situations. The proposed evaluation framework is very easy to implement. We collected words from multiple sources, domains, and languages and used the proposed evaluation framework to conduct experiments. Extensive experiments demonstrate that the proposed SpyGPT and SpyGame effectively evaluate the capabilities of various LLMs, capturing their ability to adapt to novel situations and engage in strategic communication.
BibTeX
@misc{liang2023leveraging,
  author = {Tian Liang and Zhiwei He and Jen-tse Huang and Wenxuan Wang and Wenxiang Jiao and Rui Wang and Yujiu Yang and Zhaopeng Tu and Shuming Shi and Xing Wang},
  title = {{Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models}},
  year = {2023},
  eprint = {2310.20499},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.20499},
}
Figure