TL;DR
This paper introduces CULTURAL CODES, a novel dataset that captures how sociocultural backgrounds—such as morality, personality, and demographics—influence the way humans give and interpret word-association clues in the game Codenames Duet. By incorporating these "sociocultural priors" into NLP models, the researchers demonstrate significant improvements in predicting human gameplay, proving that pragmatic inference is far from culture-neutral.
What Problem Does This Paper Address?
In communication, "pragmatics" refers to the gap between what is literally said and what is actually meant. Most current NLP models treat pragmatics as a universal, culture-free process. However, in the real world, our "common ground" is built on our specific backgrounds. For instance, the acronym "NLP" means something very different to a computer scientist than it does to a practitioner of Neuro-linguistic Programming.
The authors argue that prior work in pragmatic reasoning often uses unidentified speakers and listeners, ignoring the sociocultural factors that shape our assumptions. The challenge is to rigorously model how these diverse axes—class, age, gender, race, and values—affect interpersonal interaction and word association.
Methods
The researchers used the collaborative game Codenames Duet as a laboratory. In this game, a "Clue Giver" provides a one-word hint to help a "Guesser" identify specific target words on a board while avoiding "avoid" words.
To model this, they deconstructed gameplay into five distinct conditional generation tasks (summarized in Figure 5):
- Target Selection: Picking which words to clue.
- Clue Generation: Creating the hint.
- Clue Framing: Explaining the rationale for the clue.
- Guess Selection: Choosing words based on a hint.
- Guess Framing: Explaining the rationale for the guess.
They also included a binary classification task: Predicting Pragmatic Success (whether the guesser will correctly interpret the giver).
The core technical novelty lies in Augmenting with Sociocultural Priors. They prepended player-specific attributes (e.g., GIVER, age: 30, morality: fairness) to the input strings of standard Seq2Seq models like T5 and BART, and classifiers like RoBERTa and XLNet.
Datasets
The authors introduced the CULTURAL CODES dataset:
- Gameplay Data: 794 games (7,703 turns) played by 153 unique players recruited via Amazon Mechanical Turk.
- Player Profiles: Extensive surveys covering:
- Demographics: Age, race, education, and native language (Figure 3).
- Personality: The "Big Five" personality traits (Figure 2).
- Morality: Haidt’s Moral Foundations Theory (MFT), which measures values like Fairness, Care, and Authority (Figure 4).
- Word List: A strategically filtered subset of 100 abstract and polysemous (multi-meaning) nouns from the original Codenames to maximize the likelihood of diverse, culturally-dependent responses.
Key Insights and Ideas
- Morality is a Strong Signal: For clue-giving tasks, "Morality" priors (MFT scores) often yielded the highest performance gains. This suggests that our underlying value systems deeply influence how we associate abstract concepts.
- Specificity Over Generality: Models without sociocultural priors tend to produce "euro-centric" or generic clues (e.g., cluing
spacewithnasa). Adding priors allows models to generate more specific, culturally-tuned clues (e.g., cluingmatchwithcricket). - The Failure of "Culture-Blind" Vectors: Standard word vector baselines (like fastText) performed poorly. This indicates that pre-trained embeddings alone fail to capture the nuanced, person-to-person common ground required for successful pragmatics.
- Demographic Mismatch: Pragmatic failures (misunderstandings) were more common when there was a demographic mismatch between the giver and the guesser, highlighting the importance of shared background for successful communication.
Key Results
The inclusion of sociocultural priors universally improved model performance across all tasks.
- Clue & Target Selection: As shown in the results table (Figure 6), T5 models using Morality priors achieved the best ROUGE-1 scores for Target Selection (34.58). For Guess Selection, Demographic priors (DemoReq) were most effective (67.25).
- Pragmatic Success: In the classification task, RoBERTa’s Macro F-1 score jumped from 0.57 (no priors) to 0.67 when incorporating Personality priors.
- Qualitative Specificity: Models with priors generated more nuanced rationales. For example, instead of a generic "actors are cast," a culturally-aware model produced "actors are part of a cast."
Limitations and Future Work
- Vocabulary Constraints: The study was limited to a core vocabulary of 100 words. Future work should explore larger, more complex linguistic settings.
- Model Scale: The researchers used base-sized Seq2Seq models. Testing these tasks on Large Language Models (LLMs) like GPT-4 could reveal whether massive scale inherently captures these cultural nuances or if explicit priors are still necessary.
- Spurious Correlations: Jointly modeling all priors didn't always yield the best results, likely due to redundancies and correlations between factors like politics and morality. Better regularization techniques are needed.
- Evaluation Challenges: Automatic metrics (like ROUGE) are imperfect for cultural tasks. Finding human evaluators who perfectly match the sociocultural background of the original players is a significant logistical hurdle.
Summary updated