CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

TL;DR

Large Language Models (LLMs) often exhibit cultural biases due to their English-centric training data. This paper introduces CultureLLM, a cost-effective method to fine-tune LLMs for cultural awareness using a novel semantic data augmentation technique on the World Values Survey. CultureLLM significantly outperforms existing models, including GPT-4, on various culture-related tasks, making LLMs more equitable and useful across diverse cultural contexts.

What Problem Does This Paper Address?

Modern LLMs, trained predominantly on English-language data, tend to reflect Western cultural biases. This underrepresentation of other cultures leads to biased outputs and hinders their effective use in diverse global communities. While some approaches like prompt engineering or culture-specific pre-training exist, they are either ineffective for low-resource cultures or computationally expensive. The core challenge is to create LLMs that are culturally aware and equitable across a wide range of languages and cultures without requiring massive datasets or extensive computational resources.

Methods

CultureLLM employs a three-step approach:

  1. Sampling: A small, representative set of 50 questions from the World Values Survey (WVS) is selected as seed data. These questions cover various cultural topics like social values, migration, and political participation.
  2. Semantic Data Augmentation: This is the core innovation. The paper proposes a method to generate semantically equivalent training data. It involves:
  3. Semantic Template Generation: Using GPT-4 to rephrase seed questions into diverse sentence structures while preserving meaning. A semantic filter ensures similarity between original and generated sentences.
  4. Intact Sample Generation: Replacing words in the generated templates with context-aware synonyms to create new, semantically equivalent questions. This process aims to increase diversity at both sentence and word levels.
  5. Fine-tuning: The augmented data, along with the original seed data, is used to fine-tune LLMs. This results in either culture-specific models (e.g., CultureLLM-Ar) or a unified model (CultureLLM-One) that handles multiple cultures.

Datasets

  • Training Data: The World Values Survey (WVS) [Survey, 2022] is used as the primary seed dataset. This is augmented with synthetically generated data using the proposed semantic data augmentation method.
  • Evaluation Datasets: The paper evaluates CultureLLM on 59 culture-related downstream tasks across 9 cultures (Arabic, Bengali, Chinese, English, German, Korean, Portuguese, Spanish, Turkish). These datasets cover tasks like offensive language detection, hate speech detection, stance detection, toxicity detection, threat detection, bias detection, abusive detection, and spam detection.

Key Insights and Ideas

  • Semantic Augmentation is Key: The paper demonstrates that generating semantically equivalent questions, rather than just paraphrasing, is crucial for creating high-quality training data that improves LLM cultural awareness.
  • WVS as a Foundation: The World Values Survey, despite its age, provides a robust and diverse foundation for understanding cultural values, which can be effectively leveraged for LLM fine-tuning.
  • Cost-Effectiveness Matters: CultureLLM offers a significantly more affordable way to achieve cultural awareness in LLMs compared to traditional pre-training methods, making it accessible to a wider range of researchers.
  • Unified vs. Specific Models: While culture-specific models generally perform better, the unified CultureLLM-One model still shows substantial improvements, indicating the potential for a single model to handle diverse cultural contexts.
  • English Fine-tuning Advantage: Fine-tuning on English data (even if translated from other languages) appears to yield better results than fine-tuning directly on translated local languages, likely due to the inherent capabilities of LLMs trained on English corpora.

Key Results

  • CultureLLM significantly outperforms GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) on average across 60 culture-related datasets.
  • CultureLLM achieves performance comparable to, and in some cases exceeding, GPT-4.
  • The semantic data augmentation approach is shown to be effective, with the complete process (+WVS+a+b) yielding the best results in ablation studies.
  • A human study confirmed that the generated samples are semantically equivalent to the original ones, with a high similarity score (96.5%).
  • CultureLLM demonstrates robustness against catastrophic forgetting on general benchmarks like BIG-Bench Hard and GSM8K.

Limitations and Future Work

  • Limited Scope of Fine-tuning: The current work focuses on fine-tuning using OpenAI's API and a specific open-source model (Llama2). Future work could explore fine-tuning on a wider range of open-source LLMs.
  • Generative Tasks: While the paper evaluates on classification tasks, multilingual generative tasks were not extensively explored due to the expense of automatic evaluation. Future work could address this.
  • Sample Diversity: The current augmentation primarily focuses on sentence and word-level diversity. Future work could explore adding more diverse augmentation strategies.
  • Low-Resource Culture Performance: While CultureLLM shows improvement, performance in some low-resource cultures (like Korean) was not as pronounced, suggesting further research is needed.
  • Cultural Boundaries: The paper uses language as a proxy for culture, acknowledging that this is a simplification. Future work could explore more nuanced definitions of cultural boundaries.

Summary updated

Abstract

Large language models (LLMs) are reported to be partial to certain cultures owing to the training data dominance from the English corpora. Since multilingual cultural data are often expensive to collect, existing efforts handle this by prompt engineering or culture-specific pre-training. However, they might overlook the knowledge deficiency of low-resource culture and require extensive computing resources. In this paper, we propose CultureLLM, a cost-effective solution to incorporate cultural differences into LLMs. CultureLLM adopts World Value Survey (WVS) as seed data and generates semantically equivalent training data via the proposed semantic data augmentation. Using only 50 seed samples from WVS with augmented data, we fine-tune culture-specific LLMs and one unified model (CultureLLM-One) for 9 cultures covering rich and low-resource languages. Extensive experiments on 60 culture-related datasets demonstrate that CultureLLM significantly outperforms various counterparts such as GPT-3.5 (by 8.1%) and Gemini Pro (by 9.5%) with comparable performance to GPT-4 or even better. Our human study shows that the generated samples are semantically equivalent to the original samples, providing an effective solution for LLMs augmentation.
BibTeX
@inproceedings{li2024culturellm,
  author = {Cheng Li and Mengzhou Chen and Jindong Wang and Sunayana Sitaram and Xing Xie},
  title = {{CultureLLM: Incorporating Cultural Differences into Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=sIsbOkQmBL},
}
Figure