Abstract
Cultural evaluation of large language models has become increasingly important, yet current benchmarks often reduce culture to static facts or homogeneous values. This view conflicts with anthropological accounts that emphasize culture as dynamic, historically situated, and enacted in practice. To analyze this gap, we introduce a four-part framework that categorizes how benchmarks frame culture, such as knowledge, preference, performance, or bias. Using this lens, we qualitatively examine 20 cultural benchmarks and identify six recurring methodological issues, including treating countries as cultures, overlooking within-culture diversity, and relying on oversimplified survey formats. Drawing on established anthropological methods, we propose concrete improvements: incorporating real-world narratives and scenarios, involving cultural communities in design and validation, and evaluating models in context rather than isolation. Our aim is to guide the development of cultural benchmarks that go beyond static recall tasks and more accurately capture the responses of the models to complex cultural situations.
Introduction
As Large Language Models (LLMs) become central to global knowledge production, researchers are racing to measure their "cultural competence." However, the NLP community often operates with a "thin" definition of culture, treating it as a background variable or a fixed set of national traits. This leads to critical failures such as reductivism (treating culture as a trait-list of facts), the consensus fallacy (erasing internal diversity), and nation-state proxying (equating geographic borders with homogeneous cultures).
A Taxonomy of Cultural Framing
To bridge the gap between NLP and anthropology, we propose a four-part taxonomy for how culture is framed in current benchmarks. This framework allows researchers to identify what aspect of culture they are actually measuring:
- Culture-as-Knowledge: Factual recall of traditions, holidays, and symbols (e.g., "What do people eat in X country?").
- Culture-as-Preference: Shared moral or social values, often measured via survey-style statements.
- Culture-as-Dynamics: Culture as a "performed" entity in dialogue, narrative, and social interaction.
- Culture-as-Bias: Identifying stereotypes and preventing identity-linked harms.
Auditing the Benchmark Landscape
We performed a qualitative audit of 20 prominent cultural benchmarks, ranging from knowledge-focused datasets like CultureAtlas and SeaExam to bias-focused ones like BBQ and BOLD. Our analysis identified six recurring methodological limitations that hinder the depth of these evaluations:
- Platform Bias: Over-reliance on Western-centric sources like Reddit or Wikipedia.
- Nation-State Conflation: Using nationality as a shorthand for culture, erasing ethnic and religious complexities.
- Moral Simplification: Reducing complex ethical dilemmas to binary choices.
- The Consensus Trap: Relying on majority-vote ground truths, which inadvertently silences minority voices and flattens cultural complexity.
- Lack of Context: Stripping away the historical and situational grounding necessary for true cultural reasoning.
Rethinking Evaluation: Culture as a Verb
Drawing on anthropological theory, we argue that culture is a dynamic process of "meaning-making" rather than a static database. In anthropology, contestation and dissent are essential features, not statistical noise. Benchmarks should move toward "thick description"—evaluating how models respond to complex, context-rich scenarios rather than isolated multiple-choice questions.
Our audit revealed a significant imbalance: 19 out of 20 benchmarks exhibited platform bias, and 14 out of 20 relied on consensus. Furthermore, while "Knowledge" and "Preference" are well-represented, the "Dynamics" of situated speech remain significantly under-explored.
Conclusion and Future Directions
To build more nuanced evaluations, we call for deep collaboration with social scientists. Anthropologists should be involved throughout the pipeline, from problem definition to "Critical Annotation" that allows for ambiguity and narrative storytelling. Future work must also address multimodal cultural benchmarks and the real-world impact of these evaluation flaws on deployed applications. By moving beyond static recall tasks, the NLP community can develop models that truly understand the lived, contested, and evolving nature of human culture.
Summary updated