CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies

Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério de Paula, Diyi Yang

Abstract

To enhance language models’ cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With the pipeline, we construct CultureBank, a knowledge base built upon users’ self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit. Unlike previous cultural knowledge resources, CultureBank contains diverse views on cultural descriptors to allow flexible interpretation of cultural knowledge, and contextualized cultural scenarios to help grounded evaluation. With CultureBank, we evaluate different LLMs’ cultural awareness, and identify areas for improvement. We also fine-tune a language model on CultureBank: experiments show that it achieves better performances on two downstream cultural tasks in a zero-shot setting. Finally, we offer recommendations for future culturally aware language technologies.

Introduction

Most current Large Language Models (LLMs) are trained on vast datasets that over-represent Western perspectives, leading to significant "cultural blind spots." While formal sources like Wikipedia provide some cultural facts, they often miss the "long-tailed," evolving nuances of daily life. Furthermore, existing cultural datasets tend to be assertive, stating norms as absolute facts, whereas culture is inherently heterogeneous and often debated within groups.

To address these limitations, we introduce CultureBank, a massive, community-driven knowledge base designed to move LLMs beyond Western-centric biases. CultureBank provides a framework for "grounded evaluation"—testing if models can apply cultural nuances in realistic, conversational scenarios rather than just reciting facts.

Overview of the CultureBank framework, illustrating the transition from raw community data to culturally-aware language models.

Methodology

We developed a "bottom-up" pipeline to transform noisy social media narratives into structured, traceable knowledge. The pipeline consists of four primary stages:

  1. Extraction: A classifier identifies culturally relevant comments. An LLM (Llama-2-70B) then extracts data into a specific 11-field taxonomy, including cultural group, context, actor’s behavior, and goal.
  2. Clustering & Summarization: To manage data volume, similar behaviors are clustered using SentenceBert and Hierarchical Agglomerative Clustering (HAC). A Mixtral model then summarizes these clusters into representative descriptors.
  3. Agreement Calculation: Instead of assertive facts, the pipeline calculates an "Agreement Level" based on user consensus within a cluster, acknowledging that culture is not a monolith.
  4. Grounded Evaluation: For every descriptor, we generate a "grounded" scenario (e.g., a travel advisory) and a persona to test if a model can provide culturally aware advice in a realistic context.
The CultureBank data pipeline, showing the stages of extraction, clustering, and agreement calculation.

Datasets and Insights

CultureBank leverages two primary sources:

  • CultureBank-TikTok: 12,000 cultural descriptors across 730 cultural groups.
  • CultureBank-Reddit: 11,000 descriptors across 2,000 cultural groups, demonstrating the pipeline's generalizability.

Our approach captures nuances that formal databases miss, such as the Icelandic practice of leaving babies outside in strollers to nap in the cold. By including an agreement metric, we ensure that models understand that some practices are controversial even within their own communities. Furthermore, we found that fine-tuning models on this structured data significantly improved their performance on unrelated downstream cultural tasks, suggesting that cultural awareness is a transferable skill.

Key Results

In human head-to-head trials, our fine-tuned Mixtral-SFT model was judged more culturally aware than the base Mixtral model 58% of the time. In specific grounded scenarios, Mixtral-SFT achieved a win rate of 46% against GPT-4, with an additional 38% of cases resulting in a tie. On the GlobalOpinionQA benchmark, fine-tuning improved alignment with global human survey results, raising average similarity from 83.6 to 85.4.

Limitations and Future Work

The current pipeline is limited to English-language communities, which may miss nuances inextricably linked to native languages. Additionally, the reliance on specific hashtags (like #cultureshock) can skew data toward "surprising" rather than mundane norms. Future research will focus on multimodal cultural knowledge and temporal analysis to track how cultural norms evolve over time.

Summary updated

Abstract

To enhance language models’ cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With the pipeline, we construct CultureBank, a knowledge base built upon users’ self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit. Unlike previous cultural knowledge resources, CultureBank contains diverse views on cultural descriptors to allow flexible interpretation of cultural knowledge, and contextualized cultural scenarios to help grounded evaluation. With CultureBank, we evaluate different LLMs’ cultural awareness, and identify areas for improvement. We also fine-tune a language model on CultureBank: experiments show that it achieves better performances on two downstream cultural tasks in a zero-shot setting. Finally, we offer recommendations for future culturally aware language technologies. We release the CultureBank dataset, code and models at https://github.com/SALT-NLP/CultureBank. Our project page is at culturebank.github.io
BibTeX
@inproceedings{shi2024culturebank,
  author = {Weiyan Shi and Ryan Li and Yutong Zhang and Caleb Ziems and Sunny Yu and Raya Horesh and Rogério de Paula and Diyi Yang},
  title = {{CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=w2pp5kaOZJ},
}
Figure