Abstract
To enhance language models' cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With the pipeline, we construct CultureBank, a knowledge base built upon users' self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit. Unlike previous cultural knowledge resources, CultureBank contains diverse views on cultural descriptors to allow flexible interpretation of cultural knowledge, and contextualized cultural scenarios to help grounded evaluation. With CultureBank, we evaluate different LLMs' cultural awareness, and identify areas for improvement. We also fine-tune a language model on CultureBank: experiments show that it achieves better performances on two downstream cultural tasks in a zero-shot setting. Finally, we offer recommendations based on our findings for future culturally aware language technologies.
Introduction
Current Large Language Models (LLMs) frequently exhibit a "Western-centric" bias, a byproduct of training data that predominantly reflects Western values, behaviors, and social norms. This bias leads to models that reinforce stereotypes or provide generic, "one-size-fits-all" advice when interacting with diverse global populations.
Existing cultural knowledge bases often fall short for three primary reasons: they rely on formal sources like Wikipedia (missing the nuances of everyday life), they present cultural facts as absolute truths (ignoring internal cultural diversity), and they rely on simplistic evaluation benchmarks (like True/False questions) that do not reflect real-world conversational utility. To address these gaps, we introduce CultureBank, a massive, community-driven cultural knowledge base designed to capture "lived experiences" through social media narratives.
Methodology: The CultureBank Pipeline
We developed a "bottom-up" pipeline to transform noisy social media data into structured, actionable cultural knowledge. The process consists of five key stages:
- Extraction: Using a DistilBERT classifier, we filter for culturally relevant comments. A Llama-2-70B model then extracts these into a structured taxonomy, capturing fields such as the cultural group, context, actor’s behavior, and the level of agreement expressed in the narrative.
- Clustering & Summarization: To manage the high volume of data, we utilize SentenceBERT and Hierarchical Agglomerative Clustering (HAC) to group similar behaviors. A fine-tuned Mixtral-8x7B model then generates a representative summary for each cluster.
- Agreement Calculator: Moving beyond assertive facts, our pipeline calculates an "agreement level" based on community validation. This acknowledges that cultural practices are heterogeneous and rarely universal.
- Grounded Evaluation: We test LLMs using "grounded scenarios" generated by Mixtral (e.g., a travel consultant advising a backpacker). This evaluates whether an LLM can naturally integrate cultural knowledge into a conversation rather than just reciting facts.
- Fine-tuning: We apply Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to Llama-2 and Mixtral models to embed this cultural knowledge directly into the model weights.
The CultureBank Dataset
CultureBank is comprised of two primary components sourced from 2019–2023 data:
- CultureBank-TikTok: 12,000 cultural descriptors extracted from 720,000 comments, often utilizing hashtags like #cultureshock to identify cross-cultural observations.
- CultureBank-Reddit: 11,000 descriptors sourced from 2.6 million comments.
These datasets provide a rich, albeit noisy, alternative to formal textbooks, capturing the "long-tailed" nuances of cultural life that are often omitted from official records.
Key Insights and Results
Culture as a Non-Monolith
By incorporating an "Agreement Level," CultureBank reflects the reality that cultural practices vary. A behavior with a 0.6 agreement score indicates a common but not universal practice, allowing for more flexible and accurate cultural modeling.
The Knowledge-Application Gap
A significant finding of our research is that even when models like GPT-4 "know" a cultural fact, they often fail to apply it in grounded scenarios. Instead, they frequently fall back on generic, templated advice. Our fine-tuned models (e.g., Mixtral-8x7B-SFT) significantly improved in this area, outperforming their vanilla counterparts in cultural specificity.
Performance and Human Preference
In head-to-head human evaluations, the fine-tuned Mixtral-SFT model was judged more culturally aware than the base model 34% of the time (with 48% ties). Furthermore, models trained on CultureBank demonstrated improved zero-shot transfer on downstream tasks like GlobalOpinionQA (cross-national perspectives) and CultureNLI (cultural natural language inference), achieving distributions more similar to actual surveyed human responses across different countries.
Limitations and Future Work
While CultureBank represents a significant step forward, several limitations remain:
- Language Constraint: The current pipeline is limited to English-language social media, potentially losing nuances only expressed in native languages.
- Sampling Bias: Sourcing from #cultureshock may skew data toward "surprising" differences rather than everyday similarities.
- Modality: Future iterations aim to incorporate multimodal data (video and images) to capture gestures and rituals that text cannot describe.
- Annotator Diversity: We recommend involving a more diverse set of global human annotators in future evaluations to further reduce Western bias in the benchmarking process.
Conclusion
CultureBank demonstrates that high-quality, traceable cultural knowledge can be extracted from the chaos of social media. By focusing on lived experiences and grounded application, we provide a path toward language technologies that are truly aware of the diverse world they serve.
Summary updated