Abstract
This paper introduces CIVICS (Culturally-Informed & Values-Inclusive Corpus for Societal impacts), a hand-crafted multilingual dataset designed to evaluate the social and cultural variation of Large Language Models (LLMs). By analyzing responses to value-sensitive topics like LGBTQI rights, social welfare, and immigration across five languages, we demonstrate that LLMs exhibit significant variability in their encoded values. Our findings reveal that English prompts frequently trigger refusals and that model scaling increases "opinionatedness," with larger models taking more distinct stances. CIVICS serves as a tool for promoting transparency and developing AI that respects global cultural pluralism.
Introduction
As Large Language Models (LLMs) become central to global digital infrastructure, there is a growing concern that they primarily reflect Western-centric worldviews. Most existing benchmarks for "value alignment" are English-centric, often losing cultural nuance when machine-translated. The CIVICS project argues that values are not universal truths but are deeply embedded in language and local context. To build AI that respects global pluralism, we must measure how models behave when confronted with value-laden topics in their original cultural settings.
Methodology
The researchers moved away from automated pipelines, opting for a "human-in-the-loop" curation process. Native speakers identified 699 prompts from authoritative regional sources—such as government documents and news agencies (e.g., AFP, ANSA)—across five languages (English, French, German, Italian, and Turkish) and nine national contexts.
Evaluation utilized two distinct experimental setups:
- Log-Probability Analysis: Measuring the probability of base models predicting "Yes" vs. "No" on value statements.
- Long-Form Response Analysis: Prompting chat-tuned models for open-ended answers, which were then classified by an external model (Command-R) into categories like Agreement, Disagreement, or Refusal.
The CIVICS Dataset
The core contribution is the CIVICS dataset. Unlike previous benchmarks, CIVICS is hand-crafted by native speakers to ensure linguistic and cultural authenticity. It focuses on five ethically charged domains: LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
Key Findings
The "English Refusal" Phenomenon
Models are significantly more likely to refuse to answer a sensitive question when it is presented in English or as a translated version of a foreign statement. This suggests that "safety" guardrails might be over-tuned for English, leading to disparate performance and utility across different linguistic contexts.
Scaling and Opinionatedness
Model "personalities" shift significantly with scale. Larger models (e.g., Llama 3 70B and Yi 34B) tend to have fewer "neutral" responses than their smaller counterparts (8B and 6B). As models grow, they appear to encode more distinct—and sometimes more divisive—stances on societal issues.
Regional and Topic Variation
The study found that statements sourced from Italian contexts triggered the highest rates of disagreement across models, while English prompts generally saw the highest levels of agreement. Immigration and LGBTQI rights were the most "refusal-prone" topics, indicating that models are specifically trained to avoid taking stances on these highly polarized issues.
Key Results
In long-form experiments, Qwen was the most "cautious," refusing to answer 257 times, whereas Command-R was the most "opinionated," refusing only 8 times. Disagreement was most frequent on topics related to immigration and social welfare. The highest variation between model answers occurred on questions regarding German immigration policy and Italian LGBTQI rights.
Conclusion
CIVICS provides a vital tool for examining the cultural soul of LLMs. While it represents a snapshot of specific values, it highlights the necessity of pluralistic AI development. Future work will involve expanding the dataset to include more non-Western languages and involving a broader, more diverse group of annotators to further mitigate bias and promote global value pluralism.
Summary updated