TL;DR
Current NLP benchmarks evaluate "culture" in LLMs by reducing it to static trivia or oversimplified country-level surveys. This paper introduces an anthropological framework to critique 20 prominent cultural benchmarks, exposing six systemic flaws and offering a roadmap for designing evaluations that treat culture as a dynamic, contested, and lived experience.
What Problem Does This Paper Address?
As large language models (LLMs) are deployed globally in translation, education, and search engines, measuring their "cultural competence" has become crucial. However, current NLP benchmarks treat culture as a static background variable rather than a complex, evolving process.
Benchmark designers frequently rely on superficial proxies—like equating nationality with culture—or rigid survey formats (e.g., Likert scales). This paper bridges the gap between NLP and anthropology, arguing that current evaluation methods flatten cultural nuances, ignore within-culture diversity, and fail to capture how culture is actually enacted and negotiated in everyday life.
Methods
This position paper takes a qualitative, humanistic approach rather than performing a large-scale statistical corpus analysis. The authors curated and deeply analyzed 20 prominent cultural benchmarks in NLP.
To structure their critique, they developed a four-part taxonomy that categorizes how culture is framed within NLP:
- Culture-as-Knowledge: Factual or object-level information (e.g., holidays, traditional foods).
- Culture-as-Preference: Shared moral, social, or political values derived from aggregate survey data.
- Culture-as-Dynamics: Culture as an interactive, performative, and context-dependent phenomenon.
- Culture-as-Bias: Culture inferred from patterns of stereotyping or discriminatory output.
The authors then systematically audited the 20 benchmarks against six common methodological limitations derived from anthropological theory.
Datasets
While the paper does not introduce a new dataset, it provides a comprehensive qualitative review of 20 major cultural and bias benchmarks in NLP, including:
- Bias & Stereotyping: BBQ, BOLD, CrowS-Pairs, M-Bias, Social Bias Frames, VL-BiasBench.
- Knowledge & Fact-Checking: BLEnD, CulturalBench, CultureAtlas, SeaExam, X-FACTR.
- Values & Commonsense Reasoning: CASA, CDEval, Civics, LLM-GLOBE, MoralBench, Moral Stories, NormAd, Social Chemistry 101.
- Affective/Other: CuLEmo.
Key Insights and Ideas
- The Nation-State Fallacy: Many benchmarks treat geopolitical borders as homogeneous cultural boundaries (e.g., using statements like "In Chinese culture..."). This erases internal ethnic, linguistic, and religious diversity, and completely overlooks diasporic contexts.
- Consensus vs. Contestation: Most benchmarks use majority voting to establish a singular "ground truth," treating annotator disagreement as statistical noise. In anthropology, disagreement and negotiation are fundamental features of cultural life, not flaws to be smoothed away.
- Platform Bias: Datasets scraped from platforms like Reddit or Wikipedia inherit a heavily Western, urban, male, and digitally literate demographic skew, which is then incorrectly treated as globally representative.
- Static vs. Dynamic Framing: As illustrated in the authors' taxonomy, NLP heavily over-indexes on static Knowledge and Preference lenses, missing the performative Dynamics of how culture actually functions in dialogue and interaction.
Key Results
The authors mapped the 20 reviewed benchmarks across their proposed taxonomy to highlight the field's conceptual fragmentation:
They also documented the exact distribution of the six methodological flaws across the benchmarks, revealing widespread vulnerabilities:
- Platform Bias and Annotation Fallacies were nearly universal across the reviewed datasets.
- Moral Simplification and Consensus Assumptions heavily impacted benchmarks attempting to evaluate values and reasoning (like CASA, CDEval, and MoralBench).
The full breakdown of these limitations across all 20 benchmarks can be seen in the authors' systematic mapping table:
Limitations and Future Work
- Scope of Benchmarks: The study was restricted to 20 publicly available, widely recognized benchmarks, meaning proprietary or highly specialized datasets were not explored.
- Text-Only Focus: The analysis explicitly omitted vision-oriented and multimodal cultural benchmarks (e.g., CulturalVQA, WorldCuisine), which require entirely different frameworks for analyzing visual iconography and image-text alignment biases.
- Downstream Impact: The paper focuses purely on benchmarking methodologies rather than evaluating how these flawed benchmarks impact real-world LLM applications (like healthcare or governance).
- Future Directions: The authors advocate for a paradigm shift: establishing deeply embedded, interdisciplinary pipelines with social scientists, adopting participatory design with cultural insiders, and moving toward disagreement-aware evaluation protocols that map controversy rather than forcing artificial consensus.
Summary updated