Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariani, Fred Morstatter, Diyi Yang

Introduction

Language understanding is not a "one-size-fits-all" process. How we interpret words and clues is deeply influenced by our sociocultural backgrounds, including our age, race, values, and personality. This paper introduces CULTURAL CODES, a novel dataset based on the game Codenames Duet, to capture and model these influences. By incorporating "sociocultural priors," researchers demonstrate that AI models can significantly improve their ability to give and interpret pragmatic clues, moving closer to human-like communication.

The Challenge of Pragmatics

Pragmatics explores how context and speaker intention shape meaning beyond literal definitions. For instance, the term "NLP" might mean "Natural Language Processing" to a computer scientist but "Neuro-linguistic Programming" to a psychologist. Most current NLP models assume a generic, culture-free speaker. However, real-world communication relies on a shared "common ground" built from individual backgrounds. The authors argue that for AI to truly understand humans, it must model the specific sociocultural priors of the individuals involved.

The CULTURAL CODES Dataset

To study this, the researchers created the CULTURAL CODES dataset, consisting of 794 games of Codenames Duet (7,703 turns) played by 153 unique participants recruited via Amazon Mechanical Turk.

Distribution of political leanings and moral foundations among study participants.

For every player, the dataset includes comprehensive sociocultural metadata:

  • Demographics: Age, race, gender, education, and native language.
  • Personality: Big 5 Personality traits (Openness, Conscientiousness, etc.).
  • Morality & Politics: Self-reported political leaning and scores from the Moral Foundations Theory (MFT) survey, which measures values like Care, Fairness, Loyalty, Authority, and Sanctity.

To maximize the likelihood of diverse interpretations, the game's vocabulary was filtered to the 100 most abstract and semantically ambiguous nouns.

Modeling Framework

The researchers deconstructed Codenames Duet into five distinct modeling tasks to test the impact of sociocultural context:

The five modeling tasks derived from the Codenames Duet game.
  1. Target Selection: Choosing "goal" words to link with a clue.
  2. Clue Generation: Producing a single-word clue to connect targets while avoiding "avoid" words.
  3. Clue Framing: Explaining the rationale behind a clue-target relationship.
  4. Guess Selection: Predicting the guesser's choices based on a clue.
  5. Predicting Success: Determining if a clue-target pair will result in a successful guess.

The team used Attribute Prepending to incorporate context, converting player data into text strings (e.g., Age: 30, Political Leaning: Liberal) and adding them to the model's input. They tested architectures including T5, BART, BERT, RoBERTa, and XLNet.

Key Insights and Results

The study yielded several critical findings regarding the role of culture in language:

  • Morality as a Strong Prior: MFT scores were often the most effective features for improving clue generation. For T5, including Morality priors improved the ROUGE-1 score from 23.86 to 26.54.
  • Specificity over Genericity: Models with sociocultural priors generated more specific clues. For example, a model with priors might link "bond" to "James" (culturally specific) rather than just "connection" (generic).
  • Predicting Pragmatic Failure: Pragmatic failure often occurs due to demographic mismatches between the giver and guesser. Using Personality priors helped RoBERTa achieve a Macro F1 of 0.67 in success prediction, compared to a 0.57 baseline without priors.
  • Beyond Word Vectors: Traditional baselines like fastText performed poorly, indicating that sociocultural pragmatics requires complex reasoning about intent rather than simple word co-occurrence.

Limitations and Future Directions

While promising, the study has limitations. The vocabulary was restricted to 100 words, and the participant pool was predominantly white and North American. Future work should explore larger vocabularies and more diverse global demographics. Additionally, testing these hypotheses on larger models like GPT-4 or Llama-3 could reveal if "cultural common ground" emerges more naturally at scale. Finally, incorporating recursive pragmatic frameworks like Rational Speech Acts (RSA) might further refine performance.

Summary updated

Abstract

Pragmatic reference enables efficient interpersonal communication. Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers and listeners. In practice, however, speakers' sociocultural background shapes their pragmatic assumptions. For example, readers of this paper assume NLP refers to "Natural Language Processing," and not "Neuro-linguistic Programming." This work introduces the Cultural Codes dataset, which operationalizes sociocultural pragmatic inference in a simple word reference game. Cultural Codes is based on the multi-turn collaborative two-player game, Codenames Duet. Our dataset consists of 794 games with 7,703 turns, distributed across 153 unique players. Alongside gameplay, we collect information about players' personalities, values, and demographics. Utilizing theories of communication and pragmatics, we predict each player's actions via joint modeling of their sociocultural priors and the game context. Our experiments show that accounting for background characteristics significantly improves model performance for tasks related to both clue giving and guessing, indicating that sociocultural priors play a vital role in gameplay decisions.
BibTeX
@inproceedings{shaikh2023modeling,
  author = {Omar Shaikh and Caleb Ziems and William Held and Aryan J. Pariani and Fred Morstatter and Diyi Yang},
  title = {{Modeling Cross-Cultural Pragmatic Inference with Codenames Duet}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=HjUhLSonqm},
}
Figure