Introduction
Language understanding is not a "one-size-fits-all" process. How we interpret words and clues is deeply influenced by our sociocultural backgrounds, including our age, race, values, and personality. This paper introduces CULTURAL CODES, a novel dataset based on the game Codenames Duet, to capture and model these influences. By incorporating "sociocultural priors," researchers demonstrate that AI models can significantly improve their ability to give and interpret pragmatic clues, moving closer to human-like communication.
The Challenge of Pragmatics
Pragmatics explores how context and speaker intention shape meaning beyond literal definitions. For instance, the term "NLP" might mean "Natural Language Processing" to a computer scientist but "Neuro-linguistic Programming" to a psychologist. Most current NLP models assume a generic, culture-free speaker. However, real-world communication relies on a shared "common ground" built from individual backgrounds. The authors argue that for AI to truly understand humans, it must model the specific sociocultural priors of the individuals involved.
The CULTURAL CODES Dataset
To study this, the researchers created the CULTURAL CODES dataset, consisting of 794 games of Codenames Duet (7,703 turns) played by 153 unique participants recruited via Amazon Mechanical Turk.
For every player, the dataset includes comprehensive sociocultural metadata:
- Demographics: Age, race, gender, education, and native language.
- Personality: Big 5 Personality traits (Openness, Conscientiousness, etc.).
- Morality & Politics: Self-reported political leaning and scores from the Moral Foundations Theory (MFT) survey, which measures values like Care, Fairness, Loyalty, Authority, and Sanctity.
To maximize the likelihood of diverse interpretations, the game's vocabulary was filtered to the 100 most abstract and semantically ambiguous nouns.
Modeling Framework
The researchers deconstructed Codenames Duet into five distinct modeling tasks to test the impact of sociocultural context:
- Target Selection: Choosing "goal" words to link with a clue.
- Clue Generation: Producing a single-word clue to connect targets while avoiding "avoid" words.
- Clue Framing: Explaining the rationale behind a clue-target relationship.
- Guess Selection: Predicting the guesser's choices based on a clue.
- Predicting Success: Determining if a clue-target pair will result in a successful guess.
The team used Attribute Prepending to incorporate context, converting player data into text strings (e.g., Age: 30, Political Leaning: Liberal) and adding them to the model's input. They tested architectures including T5, BART, BERT, RoBERTa, and XLNet.
Key Insights and Results
The study yielded several critical findings regarding the role of culture in language:
- Morality as a Strong Prior: MFT scores were often the most effective features for improving clue generation. For T5, including Morality priors improved the ROUGE-1 score from 23.86 to 26.54.
- Specificity over Genericity: Models with sociocultural priors generated more specific clues. For example, a model with priors might link "bond" to "James" (culturally specific) rather than just "connection" (generic).
- Predicting Pragmatic Failure: Pragmatic failure often occurs due to demographic mismatches between the giver and guesser. Using Personality priors helped RoBERTa achieve a Macro F1 of 0.67 in success prediction, compared to a 0.57 baseline without priors.
- Beyond Word Vectors: Traditional baselines like fastText performed poorly, indicating that sociocultural pragmatics requires complex reasoning about intent rather than simple word co-occurrence.
Limitations and Future Directions
While promising, the study has limitations. The vocabulary was restricted to 100 words, and the participant pool was predominantly white and North American. Future work should explore larger vocabularies and more diverse global demographics. Additionally, testing these hypotheses on larger models like GPT-4 or Llama-3 could reveal if "cultural common ground" emerges more naturally at scale. Finally, incorporating recursive pragmatic frameworks like Rational Speech Acts (RSA) might further refine performance.
Summary updated