CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

TL;DR

CultureLLM is a highly cost-effective framework designed to inject cultural awareness into Large Language Models (LLMs) using just 50 seed questions from the World Values Survey. By leveraging a novel semantic data augmentation pipeline and low-cost fine-tuning, CultureLLM significantly outperforms GPT-3.5 and Gemini Pro on culture-related downstream tasks, achieving parity with or even exceeding GPT-4 for a fraction of the cost.

What Problem Does This Paper Address?

Most state-of-the-art LLMs suffer from a distinct Western-centric bias because their training data is overwhelmingly dominated by English-language corpora. This leaves low-resource cultures severely underrepresented.

When developers try to fix this, they usually turn to two strategies: prompt engineering or culture-specific pre-training. However, prompt engineering often fails because the model simply lacks the underlying cultural knowledge to begin with. Meanwhile, pre-training or fine-tuning on massive, culturally diverse datasets is incredibly expensive, computationally heavy, and practically impossible for low-resource languages where data is scarce. CultureLLM tackles this bottleneck by providing an affordable, data-efficient way to teach LLMs about global cultural nuances.

Methods

The authors introduce CultureLLM, a three-step pipeline consisting of sampling, semantic data augmentation, and fine-tuning.

Figure 1: Overview of CultureLLM. CultureLLM consists of three steps: sampling, semantic data augmentation, and fine-tuning. Culture-specific and unified CultureLLM can be fine-tuned.
  1. Sampling: The authors manually select 50 seed questions from the World Values Survey (WVS), covering seven core topics like social values, religion, and political participation. They pair these questions with survey-derived ground-truth opinions from different cultures.
  2. Semantic Data Augmentation: To expand this tiny dataset without altering the underlying cultural meaning, they propose a two-stage augmentation process:
  3. Semantic Template Generation: GPT-4 rephrases the seed questions into stylistically diverse sentences. A BERT-based semantic preservation filter removes any generations that drift too far in meaning, and NLTK parses the remaining sentences to create templates with replaceable word slots.
  4. Intact Sample Generation: GPT-4 fills those slots with context-aware synonyms, creating a diverse pool of training prompts that all share the exact same semantic meaning as the original seed question.
  5. Fine-tuning: Finally, they fine-tune the base model (e.g., GPT-3.5 or Llama-2) on the combined seed and augmented data. They train both culture-specific models (e.g., CultureLLM-Ar for Arabic) and a unified model (CultureLLM-One) that uses system prompts to switch between 9 different cultures.
Figure 1: Overview of CultureLLM. CultureLLM consists of three steps: sampling, semantic data augmentation, and fine-tuning. Culture-specific and unified CultureLLM can be fine-tuned.

Datasets

  • Training Seed Data: 50 questions selected from the World Values Survey (WVS-7).
  • Evaluation Data: 59 culture-related public datasets spanning 9 distinct cultures/languages (Arabic, Bengali, Chinese, English, German, Korean, Portuguese, Spanish, and Turkish), totaling 68,672 test samples. These datasets cover binary and multi-class classification tasks such as hate speech detection, stance detection, toxicity detection, and bias detection.
  • Open-Ended Generation Dataset: A newly constructed dataset of 65 open-ended questions used to evaluate the models' generative capabilities.

Key Insights and Ideas

  • The Power of Tiny Seeds: You don't need gigabytes of data to shift an LLM's cultural perspective. Fifty well-chosen questions, when combined with robust semantic expansion, are enough to teach a model the baseline values of a specific culture.
  • English as a Universal Scaffold: Interestingly, the WVS data used was entirely in English. The authors found that fine-tuning models on English cultural data and relying on cross-lingual transfer actually outperformed fine-tuning on data translated into local languages.
  • More Data Isn't Always Better: During ablation studies, the authors noted that performance peaked at 500 augmented samples. Pushing past 1,000 samples caused overfitting and a decline in performance, proving that controlled diversity matters more than raw volume.
  • Immunity to Catastrophic Forgetting: Fine-tuning on these niche cultural datasets did not degrade the models' general capabilities; performance on standard benchmarks like Big-Bench Hard (BBH) and GSM8K remained stable or even slightly improved.

Key Results

  • Performance Gains: CultureLLM (built on GPT-3.5) outperformed vanilla GPT-3.5 by 8.1% and Gemini Pro by 9.5% on average F1 scores across the downstream tasks.
  • GPT-4 Competitor: It achieved performance comparable to or occasionally better than GPT-4, particularly on high-resource language tasks.
  • Cost Efficiency: While traditional pre-training costs thousands of dollars, fine-tuning a CultureLLM model via the OpenAI API costs roughly $6.
Figure 3: (a) The main results averaged by cultures (left) and by tasks (right). Both CultureLLM and CultureLLM-One significantly outperform CultureLLM and Gemini with CultureLLM achieving the best performance comparable to GPT-4. (b) Ablation study. ‘+WVS’ denotes the fine-tuned models using only the 50 samples from WVS, ‘+WVS+a’ denotes fine-tuning using the WVS samples and the generated samples in step 1 of our data augmentation (i.e., using only GPT-4 to generate), and ‘+WVS+a+b’ denotes the complete process of our algorithm.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Classification Bias in Evaluation: Due to the high cost of automatically evaluating open-ended text, the benchmarking was heavily weighted toward classification tasks rather than complex generative tasks.
  • Surface-Level Diversity: The data augmentation approach is currently limited to sentence- and word-level paraphrasing.
  • Scale Constraints: Due to resource limitations, the framework has not yet been extensively tested on ultra-large-scale open-source models. Future work will focus on introducing deeper structural and conceptual diversity into the data generation pipeline.

Summary updated

Abstract

Large language models (LLMs) are reported to be partial to certain cultures owing to the training data dominance from the English corpora. Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training. However, they might overlook the knowledge deficiency of low-resource culture and require extensive computing resources. In this paper, we propose CultureLLM, a cost-effective solution to incorporate cultural differences into LLMs. CultureLLM adopts World Value Survey (WVS) as seed data and generates semantically equivalent training data via the proposed semantic data augmentation. Using only 50 seed samples from WVS with augmented data, we fine-tune culture-specific LLMs and one unified model (CultureLLM-One) for 9 cultures covering rich and low-resource languages. Extensive experiments on 60 culture-related datasets demonstrate that CultureLLM significantly outperforms various counterparts such as GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable performance to GPT-4 or even better. Our human study shows that the generated samples are semantically equivalent to the original samples, providing an effective solution for LLMs augmentation. Code is released at https://github.com/Scarelette/CultureLLM.
BibTeX
@inproceedings{li2024culturellm,
  author = {Cheng Li and Mengzhou Chen and Jindong Wang and Sunayana Sitaram and Xing Xie},
  title = {{CultureLLM: Incorporating Cultural Differences into Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=sIsbOkQmBL},
}
Figure