Towards Measuring and Modeling "Culture" in LLMs: A Survey

Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O'Neill, Ashutosh Modi, Monojit Choudhury

As Large Language Models (LLMs) become foundational to global digital infrastructure, the question of their "cultural intelligence" has moved from a theoretical concern to a high-stakes engineering challenge. Most state-of-the-art models are trained on datasets that are disproportionately Western and Anglo-centric, leading to significant performance gaps in non-Western contexts and the potential for cultural "erasure" through homogeneity. This survey provides a comprehensive map of the field, analyzing over 90 papers published between 2020 and 2024 to understand how researchers define, measure, and model culture.

A Taxonomy of Cultural Proxies

A central finding of the survey is the "definition void": none of the analyzed papers explicitly defined "culture." Instead, researchers rely on measurable stand-ins, or proxies. The authors propose a taxonomy dividing these into Demographic Proxies—such as Region, Language, and Religion—and Semantic Proxies—such as Values, Food, and Social Relations.

Taxonomy of Demographic and Semantic Proxies used in cultural LLM research.

Geographical region and language remain the dominant proxies, used in 37 and 35 papers respectively. However, semantic coverage remains narrow; while there are many recognized semantic domains (like kinship or technology), the majority of studies focus almost exclusively on emotions and values.

Probing Methodologies

The survey also categorizes the methods used to "interrogate" these models. The vast majority of studies employ Black-box Probing, which evaluates text outputs through generative or discriminative prompts. In contrast, White-box Probing—which seeks to understand how culture is mechanistically encoded within model weights—is significantly underrepresented, with only one study in the survey attempting this approach.

Taxonomy of probing methods for evaluating cultural knowledge in LLMs.

Key Insights and Challenges

The review highlights several critical gaps in current research:

  • Thin vs. Thick Descriptions: Borrowing from anthropology, the authors argue that NLP currently offers "thin" descriptions (observing surface-level outputs) rather than "thick" descriptions that capture the context and intent behind cultural behavior.
  • The "Aboutness" Gap: While research often examines how a model speaks (e.g., sentiment or language), there is little focus on what it speaks about—the local news, history, and priorities that define a culture's "aboutness."
  • Methodological Fragility: Cultural probing is highly sensitive to prompt wording; a model may appear culturally aware with one prompt and biased with a slightly rephrased version.

Future Directions

The authors conclude by calling for a more interdisciplinary approach, urging NLP researchers to collaborate with sociologists and anthropologists. Future work must move beyond simple translation-based multilingualism to capture unique cultural concepts and develop "situated" studies that evaluate how LLM biases impact real-world communities, particularly in the Global South.

Summary updated

BibTeX
@inproceedings{adilazuarda2024towards,
  author = {Muhammad Farid Adilazuarda and Sagnik Mukherjee and Pradhyumna Lavania and Siddhant Singh and Ashutosh Dwivedi and Alham Fikri Aji and Jacki O'Neill and Ashutosh Modi and Monojit Choudhury},
  title = {{Towards Measuring and Modeling "Culture" in LLMs: A Survey}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.882},
}
Figure