CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models

Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, Margaret Mitchell

Abstract

This paper introduces CIVICS (Culturally-Informed & Values-Inclusive Corpus for Societal impacts), a hand-crafted multilingual dataset designed to evaluate the social and cultural variation of Large Language Models (LLMs). By analyzing responses to value-sensitive topics like LGBTQI rights, social welfare, and immigration across five languages, we demonstrate that LLMs exhibit significant variability in their encoded values. Our findings reveal that English prompts frequently trigger refusals and that model scaling increases "opinionatedness," with larger models taking more distinct stances. CIVICS serves as a tool for promoting transparency and developing AI that respects global cultural pluralism.

Introduction

As Large Language Models (LLMs) become central to global digital infrastructure, there is a growing concern that they primarily reflect Western-centric worldviews. Most existing benchmarks for "value alignment" are English-centric, often losing cultural nuance when machine-translated. The CIVICS project argues that values are not universal truths but are deeply embedded in language and local context. To build AI that respects global pluralism, we must measure how models behave when confronted with value-laden topics in their original cultural settings.

Methodology

The researchers moved away from automated pipelines, opting for a "human-in-the-loop" curation process. Native speakers identified 699 prompts from authoritative regional sources—such as government documents and news agencies (e.g., AFP, ANSA)—across five languages (English, French, German, Italian, and Turkish) and nine national contexts.

Evaluation utilized two distinct experimental setups:

  1. Log-Probability Analysis: Measuring the probability of base models predicting "Yes" vs. "No" on value statements.
  2. Long-Form Response Analysis: Prompting chat-tuned models for open-ended answers, which were then classified by an external model (Command-R) into categories like Agreement, Disagreement, or Refusal.

The CIVICS Dataset

The core contribution is the CIVICS dataset. Unlike previous benchmarks, CIVICS is hand-crafted by native speakers to ensure linguistic and cultural authenticity. It focuses on five ethically charged domains: LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.

Key Findings

The "English Refusal" Phenomenon

Models are significantly more likely to refuse to answer a sensitive question when it is presented in English or as a translated version of a foreign statement. This suggests that "safety" guardrails might be over-tuned for English, leading to disparate performance and utility across different linguistic contexts.

Scaling and Opinionatedness

Model "personalities" shift significantly with scale. Larger models (e.g., Llama 3 70B and Yi 34B) tend to have fewer "neutral" responses than their smaller counterparts (8B and 6B). As models grow, they appear to encode more distinct—and sometimes more divisive—stances on societal issues.

Comparison of model responses across sizes and languages for disability rights and social welfare. Larger models (right columns) show more variation and a lower proportion of "neutral" labels compared to smaller models (left columns).

Regional and Topic Variation

The study found that statements sourced from Italian contexts triggered the highest rates of disagreement across models, while English prompts generally saw the highest levels of agreement. Immigration and LGBTQI rights were the most "refusal-prone" topics, indicating that models are specifically trained to avoid taking stances on these highly polarized issues.

Proportion of responses for the Yi-34B model on disability rights across five languages, showing significant variation in agreement and disagreement levels.

Key Results

In long-form experiments, Qwen was the most "cautious," refusing to answer 257 times, whereas Command-R was the most "opinionated," refusing only 8 times. Disagreement was most frequent on topics related to immigration and social welfare. The highest variation between model answers occurred on questions regarding German immigration policy and Italian LGBTQI rights.

Conclusion

CIVICS provides a vital tool for examining the cultural soul of LLMs. While it represents a snapshot of specific values, it highlights the necessity of pluralistic AI development. Future work will involve expanding the dataset to include more non-Western languages and involving a broader, more diverse group of annotators to further mitigate bias and promote global value pluralism.

Summary updated

Abstract

This paper introduces the "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset, designed to evaluate the social and cultural variation of Large Language Models (LLMs) across multiple languages and value-sensitive topics. We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy. CIVICS is designed to generate responses showing LLMs' encoded and implicit values. Through our dynamic annotation processes, tailored prompt design, and experiments, we investigate how open-weight LLMs respond to value-sensitive issues, exploring their behavior across diverse linguistic and cultural contexts. Using two experimental set-ups based on log-probabilities and long-form responses, we show social and cultural variability across different LLMs. Specifically, experiments involving long-form responses demonstrate that refusals are triggered disparately across models, but consistently and more frequently in English or translated statements. Moreover, specific topics and sources lead to more pronounced differences across model answers, particularly on immigration, LGBTQI rights, and social welfare. As shown by our experiments, the CIVICS dataset aims to serve as a tool for future research, promoting reproducibility and transparency across broader linguistic settings, and furthering the development of AI technologies that respect and reflect global cultural diversities and value pluralism. The CIVICS dataset and tools will be made available upon publication under open licenses; an anonymized version is currently available at https://huggingface.co/CIVICS-dataset.
BibTeX
@inproceedings{pistilli2024civics,
  author = {Giada Pistilli and Alina Leidinger and Yacine Jernite and Atoosa Kasirzadeh and Alexandra Sasha Luccioni and Margaret Mitchell},
  title = {{CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models}},
  booktitle = {AIES},
  year = {2024},
  doi = {10.1609/aies.v7i1.31710},
}
Figure