Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens

Mai Alkhamissi, Yunze Xiao, Badr AlKhamissi, Mona T. Diab

TL;DR

Current NLP benchmarks evaluate "culture" in LLMs by reducing it to static trivia or oversimplified country-level surveys. This paper introduces an anthropological framework to critique 20 prominent cultural benchmarks, exposing six systemic flaws and offering a roadmap for designing evaluations that treat culture as a dynamic, contested, and lived experience.

What Problem Does This Paper Address?

As large language models (LLMs) are deployed globally in translation, education, and search engines, measuring their "cultural competence" has become crucial. However, current NLP benchmarks treat culture as a static background variable rather than a complex, evolving process.

Benchmark designers frequently rely on superficial proxies—like equating nationality with culture—or rigid survey formats (e.g., Likert scales). This paper bridges the gap between NLP and anthropology, arguing that current evaluation methods flatten cultural nuances, ignore within-culture diversity, and fail to capture how culture is actually enacted and negotiated in everyday life.

Methods

This position paper takes a qualitative, humanistic approach rather than performing a large-scale statistical corpus analysis. The authors curated and deeply analyzed 20 prominent cultural benchmarks in NLP.

To structure their critique, they developed a four-part taxonomy that categorizes how culture is framed within NLP:

  1. Culture-as-Knowledge: Factual or object-level information (e.g., holidays, traditional foods).
  2. Culture-as-Preference: Shared moral, social, or political values derived from aggregate survey data.
  3. Culture-as-Dynamics: Culture as an interactive, performative, and context-dependent phenomenon.
  4. Culture-as-Bias: Culture inferred from patterns of stereotyping or discriminatory output.

The authors then systematically audited the 20 benchmarks against six common methodological limitations derived from anthropological theory.

Datasets

While the paper does not introduce a new dataset, it provides a comprehensive qualitative review of 20 major cultural and bias benchmarks in NLP, including:

  • Bias & Stereotyping: BBQ, BOLD, CrowS-Pairs, M-Bias, Social Bias Frames, VL-BiasBench.
  • Knowledge & Fact-Checking: BLEnD, CulturalBench, CultureAtlas, SeaExam, X-FACTR.
  • Values & Commonsense Reasoning: CASA, CDEval, Civics, LLM-GLOBE, MoralBench, Moral Stories, NormAd, Social Chemistry 101.
  • Affective/Other: CuLEmo.

Key Insights and Ideas

  • The Nation-State Fallacy: Many benchmarks treat geopolitical borders as homogeneous cultural boundaries (e.g., using statements like "In Chinese culture..."). This erases internal ethnic, linguistic, and religious diversity, and completely overlooks diasporic contexts.
  • Consensus vs. Contestation: Most benchmarks use majority voting to establish a singular "ground truth," treating annotator disagreement as statistical noise. In anthropology, disagreement and negotiation are fundamental features of cultural life, not flaws to be smoothed away.
  • Platform Bias: Datasets scraped from platforms like Reddit or Wikipedia inherit a heavily Western, urban, male, and digitally literate demographic skew, which is then incorrectly treated as globally representative.
  • Static vs. Dynamic Framing: As illustrated in the authors' taxonomy, NLP heavily over-indexes on static Knowledge and Preference lenses, missing the performative Dynamics of how culture actually functions in dialogue and interaction.
Cultural Framing Taxonomy

Key Results

The authors mapped the 20 reviewed benchmarks across their proposed taxonomy to highlight the field's conceptual fragmentation:

Mapping Benchmarks to Taxonomy

They also documented the exact distribution of the six methodological flaws across the benchmarks, revealing widespread vulnerabilities:

  • Platform Bias and Annotation Fallacies were nearly universal across the reviewed datasets.
  • Moral Simplification and Consensus Assumptions heavily impacted benchmarks attempting to evaluate values and reasoning (like CASA, CDEval, and MoralBench).

The full breakdown of these limitations across all 20 benchmarks can be seen in the authors' systematic mapping table:

Methodological Limitations Table

Limitations and Future Work

  • Scope of Benchmarks: The study was restricted to 20 publicly available, widely recognized benchmarks, meaning proprietary or highly specialized datasets were not explored.
  • Text-Only Focus: The analysis explicitly omitted vision-oriented and multimodal cultural benchmarks (e.g., CulturalVQA, WorldCuisine), which require entirely different frameworks for analyzing visual iconography and image-text alignment biases.
  • Downstream Impact: The paper focuses purely on benchmarking methodologies rather than evaluating how these flawed benchmarks impact real-world LLM applications (like healthcare or governance).
  • Future Directions: The authors advocate for a paradigm shift: establishing deeply embedded, interdisciplinary pipelines with social scientists, adopting participatory design with cultural insiders, and moving toward disagreement-aware evaluation protocols that map controversy rather than forcing artificial consensus.

Summary updated

Abstract

Cultural evaluation of large language models has become increasingly important, yet current benchmarks often reduce culture to static facts or homogeneous values. This view conflicts with anthropological accounts that emphasize culture as dynamic, historically situated, and enacted in practice. To analyze this gap, we introduce a four-part framework that categorizes how benchmarks frame culture, such as knowledge, preference, performance, or bias. Using this lens, we qualitatively examine 20 cultural benchmarks and identify six recurring methodological issues, including treating countries as cultures, overlooking within-culture diversity, and relying on oversimplified survey formats. Drawing on established anthropological methods, we propose concrete improvements: incorporating real-world narratives and scenarios, involving cultural communities in design and validation, and evaluating models in context rather than isolation. Our aim is to guide the development of cultural benchmarks that go beyond static recall tasks and more accurately capture the responses of the models to complex cultural situations.
BibTeX
@inproceedings{alkhamissi2026hire,
  author = {Mai Alkhamissi and Yunze Xiao and Badr AlKhamissi and Mona T. Diab},
  title = {{Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lens}},
  booktitle = {EACL (Findings)},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.63},
}
Figure