Finding deceivers in social context with large language models and how to find them: the case of the Mafia game

B. Y. Yoo, Kyung-Joong KimScientific 2024 · DOI

TL;DR

Researchers have successfully used GPT-4 to unmask "mafiosi" in the social deduction game Mafia, significantly outperforming both previous BERT-based models and human players. By leveraging zero-shot Chain-of-Thought reasoning, the LLM acted as an impartial observer, proving that as social complexity and the volume of dialogue increase, AI remains robust while human judgment often falters.

What Problem Does This Paper Address?

Detecting malicious deception in real-world social contexts is notoriously difficult. People rarely self-report their lies, and gathering high-quality data on intentional, exploitative deception is a major hurdle for behavioral science.

The authors turn to social deduction games (like Mafia or Werewolf) as a proxy. These games provide a "closed-world" environment where deception is a core mechanic, roles are pre-defined, and every interaction is recorded. Previous attempts to automate "mafia" detection relied on smaller models like BERT, which often struggled with the nuances of long-form dialogue or lacked the reasoning capabilities to connect disparate clues across multiple "game days." The challenge was to see if Large Language Models (LLMs) could use their superior reasoning and linguistic understanding to spot deceivers better than humans can.

Methods

The study compared several models: GPT-4, GPT-3.5-turbo, and two fine-tuned BERT variants (Multilabel and Utterance-based).

The researchers employed a "pure observer" approach. The LLMs were fed anonymized transcripts of real human Mafia games. Crucially, they used Prompt Engineering with a Zero-shot Chain-of-Thought (CoT) technique. By including the instruction "Write your reason step-by-step," the authors forced the model to articulate its logic before naming suspects.

The models were tested on "partial information" scenarios—meaning they only saw the game up to a specific point (e.g., the end of Day 2 or Day 3)—to mimic the real-time uncertainty players face.

Datasets

The study utilized a publicly available Mafia game dataset (Ibraheem et al., 2022) consisting of dialogues from 460 English-speaking participants recruited via Amazon Mechanical Turk.

  • Training Set: 23 games (used primarily to fine-tune the BERT baselines).
  • Test Set: 15 games.
  • Data Format: Transcripts were anonymized (e.g., P0, P1) and included both conversational text and "system info" (who was killed at night, who voted for whom).

Key Insights and Ideas

  • The "Observer" Advantage: Unlike human players who are "inside" the game and subject to social pressure, suspicion, and cognitive overload, the LLM acts as a "pure observer." This makes it more robust against "public opinion control" or "gaslighting" by the mafia.
  • Memory vs. Processing: Humans act based on fallible memory. As the game gets longer, humans find it harder to track every utterance. Conversely, GPT-4 processes the entire history at once, allowing it to spot inconsistencies across multiple days that humans might forget.
  • Diverging Accuracy Trends: One of the most striking findings was that as more information became available (later game days), GPT-4’s accuracy increased, while human accuracy actually decreased. This suggests that humans are easily swayed by "late-game" lies, whereas the AI becomes more certain as the evidence pile grows.
  • The Necessity of Dialogue: An ablation study revealed that while voting patterns are the strongest signal, removing the dialogue significantly hurt performance. This proves that "linguistic cues"—how people defend themselves or accuse others—are vital for high-level deception detection.

Key Results

  • Single-Match Accuracy: GPT-4 correctly identified at least one of the two mafiosi 80.65% of the time. This crushed the second-best model (BERT-Utterance at 56.13%) and GPT-3.5-turbo (41.94%).
  • Exact-Match Accuracy: GPT-4 correctly identified both mafiosi in 19.35% of cases. While this sounds low, GPT-3.5 and BERT-Multilabel failed to get a single exact match (0%).
  • AI vs. Human: In a head-to-head comparison, GPT-4 achieved 45.16% accuracy in predicting a single mafia member, compared to just 28.83% for human participants.

Limitations and Future Work

The authors are transparent about several hurdles:

  1. Data Shortage: The test set was small (15 games), which limits the statistical power of the late-game results.
  2. Logic Hallucinations: While GPT-4 provided "reasons," they weren't always sound. Sometimes the model correctly identified a deceiver but used flawed logic or misunderstood a game rule (e.g., misinterpreting the term "execution").
  3. Model Diversity: The study focused on OpenAI's proprietary models. Future work should explore open-source models like LLaMA, which could be fine-tuned specifically for social deduction.
  4. Complexity: The researchers hope to move beyond the "observer" role and test LLMs as active players in more complex social scenarios with more roles and rules.

Summary updated

Abstract

Lies are ubiquitous and often happen in social interactions. However, socially conducted deceptions make it hard to get data since people are unlikely to self-report their intentional deception behaviors, especially malicious ones. Social deduction games, a type of social game where deception is a key gameplay mechanic, can be a good alternative to studying social deceptions. Hence, we utilized large language models' (LLMs) high performance in solving complex scenarios that require reasoning and prompt engineering to detect deceivers in the game of Mafia given only partial information and found such an approach acquired better accuracy than previous BERT-based methods in human data and even surpassed human accuracy. Furthermore, we conducted extensive experiments and analyses to find out the strategies behind LLM's reasoning process so that humans could understand the gist of LLM's strategy.
BibTeX
@inproceedings{yoo2024finding,
  author = {B. Y. Yoo and Kyung-Joong Kim},
  title = {{Finding deceivers in social context with large language models and how to find them: the case of the Mafia game}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-81997-5},
}
Figure