Modeling Cross-Cultural Pragmatic Inference with Codenames Duet

Omar Shaikh, Caleb Ziems, William Held, Aryan J. Pariani, Fred Morstatter, Diyi Yang

TL;DR

This paper introduces CULTURAL CODES, a novel dataset that captures how sociocultural backgrounds—such as morality, personality, and demographics—influence the way humans give and interpret word-association clues in the game Codenames Duet. By incorporating these "sociocultural priors" into NLP models, the researchers demonstrate significant improvements in predicting human gameplay, proving that pragmatic inference is far from culture-neutral.

What Problem Does This Paper Address?

In communication, "pragmatics" refers to the gap between what is literally said and what is actually meant. Most current NLP models treat pragmatics as a universal, culture-free process. However, in the real world, our "common ground" is built on our specific backgrounds. For instance, the acronym "NLP" means something very different to a computer scientist than it does to a practitioner of Neuro-linguistic Programming.

The authors argue that prior work in pragmatic reasoning often uses unidentified speakers and listeners, ignoring the sociocultural factors that shape our assumptions. The challenge is to rigorously model how these diverse axes—class, age, gender, race, and values—affect interpersonal interaction and word association.

Methods

The researchers used the collaborative game Codenames Duet as a laboratory. In this game, a "Clue Giver" provides a one-word hint to help a "Guesser" identify specific target words on a board while avoiding "avoid" words.

To model this, they deconstructed gameplay into five distinct conditional generation tasks (summarized in Figure 5):

  1. Target Selection: Picking which words to clue.
  2. Clue Generation: Creating the hint.
  3. Clue Framing: Explaining the rationale for the clue.
  4. Guess Selection: Choosing words based on a hint.
  5. Guess Framing: Explaining the rationale for the guess.

They also included a binary classification task: Predicting Pragmatic Success (whether the guesser will correctly interpret the giver).

The core technical novelty lies in Augmenting with Sociocultural Priors. They prepended player-specific attributes (e.g., GIVER, age: 30, morality: fairness) to the input strings of standard Seq2Seq models like T5 and BART, and classifiers like RoBERTa and XLNet.

Figure 1: An example interaction where difference in sociocultural background results in misinterpreta- tion. Steps 1-5 outline high-level gameplay tasks. THE CLUE GIVER targets the words fall and drop, giving the hint slip. THE GUESSER misinterprets slip as a piece of paper, guessing reciept and check.

Datasets

The authors introduced the CULTURAL CODES dataset:

  • Gameplay Data: 794 games (7,703 turns) played by 153 unique players recruited via Amazon Mechanical Turk.
  • Player Profiles: Extensive surveys covering:
  • Demographics: Age, race, education, and native language (Figure 3).
  • Personality: The "Big Five" personality traits (Figure 2).
  • Morality: Haidt’s Moral Foundations Theory (MFT), which measures values like Fairness, Care, and Authority (Figure 4).
  • Word List: A strategically filtered subset of 100 abstract and polysemous (multi-meaning) nouns from the original Codenames to maximize the likelihood of diverse, culturally-dependent responses.
Figure 4: Self-reported political leaning (left) and Haidt and Graham (2007)’s Moral Foundations The- ory (right) across annotators. A majority of our work- ers are liberal (57%), 39% are conservative, and the remaining 5% are libertarian. As observed in Haidt (2012), values like loyalty, authority, and sanctity are higher for conservative leaning annotators, while fair- ness is higher for liberal annotators (p < 0.05, t-test)

Key Insights and Ideas

  • Morality is a Strong Signal: For clue-giving tasks, "Morality" priors (MFT scores) often yielded the highest performance gains. This suggests that our underlying value systems deeply influence how we associate abstract concepts.
  • Specificity Over Generality: Models without sociocultural priors tend to produce "euro-centric" or generic clues (e.g., cluing space with nasa). Adding priors allows models to generate more specific, culturally-tuned clues (e.g., cluing match with cricket).
  • The Failure of "Culture-Blind" Vectors: Standard word vector baselines (like fastText) performed poorly. This indicates that pre-trained embeddings alone fail to capture the nuanced, person-to-person common ground required for successful pragmatics.
  • Demographic Mismatch: Pragmatic failures (misunderstandings) were more common when there was a demographic mismatch between the giver and the guesser, highlighting the importance of shared background for successful communication.

Key Results

The inclusion of sociocultural priors universally improved model performance across all tasks.

  • Clue & Target Selection: As shown in the results table (Figure 6), T5 models using Morality priors achieved the best ROUGE-1 scores for Target Selection (34.58). For Guess Selection, Demographic priors (DemoReq) were most effective (67.25).
  • Pragmatic Success: In the classification task, RoBERTa’s Macro F-1 score jumped from 0.57 (no priors) to 0.67 when incorporating Personality priors.
  • Qualitative Specificity: Models with priors generated more nuanced rationales. For example, instead of a generic "actors are cast," a culturally-aware model produced "actors are part of a cast."
Quantitative results for Target and Guess selection.

Limitations and Future Work

  • Vocabulary Constraints: The study was limited to a core vocabulary of 100 words. Future work should explore larger, more complex linguistic settings.
  • Model Scale: The researchers used base-sized Seq2Seq models. Testing these tasks on Large Language Models (LLMs) like GPT-4 could reveal whether massive scale inherently captures these cultural nuances or if explicit priors are still necessary.
  • Spurious Correlations: Jointly modeling all priors didn't always yield the best results, likely due to redundancies and correlations between factors like politics and morality. Better regularization techniques are needed.
  • Evaluation Challenges: Automatic metrics (like ROUGE) are imperfect for cultural tasks. Finding human evaluators who perfectly match the sociocultural background of the original players is a significant logistical hurdle.

Summary updated

Abstract

Pragmatic reference enables efficient interpersonal communication. Prior work uses simple reference games to test models of pragmatic reasoning, often with unidentified speakers and listeners. In practice, however, speakers' sociocultural background shapes their pragmatic assumptions. For example, readers of this paper assume NLP refers to "Natural Language Processing," and not "Neuro-linguistic Programming." This work introduces the Cultural Codes dataset, which operationalizes sociocultural pragmatic inference in a simple word reference game. Cultural Codes is based on the multi-turn collaborative two-player game, Codenames Duet. Our dataset consists of 794 games with 7,703 turns, distributed across 153 unique players. Alongside gameplay, we collect information about players' personalities, values, and demographics. Utilizing theories of communication and pragmatics, we predict each player's actions via joint modeling of their sociocultural priors and the game context. Our experiments show that accounting for background characteristics significantly improves model performance for tasks related to both clue giving and guessing, indicating that sociocultural priors play a vital role in gameplay decisions.
BibTeX
@inproceedings{shaikh2023modeling,
  author = {Omar Shaikh and Caleb Ziems and William Held and Aryan J. Pariani and Fred Morstatter and Diyi Yang},
  title = {{Modeling Cross-Cultural Pragmatic Inference with Codenames Duet}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=HjUhLSonqm},
}
Figure