As Large Language Models (LLMs) become foundational to global digital infrastructure, the question of their "cultural intelligence" has moved from a theoretical concern to a high-stakes engineering challenge. Most state-of-the-art models are trained on datasets that are disproportionately Western and Anglo-centric, leading to significant performance gaps in non-Western contexts and the potential for cultural "erasure" through homogeneity. This survey provides a comprehensive map of the field, analyzing over 90 papers published between 2020 and 2024 to understand how researchers define, measure, and model culture.
A Taxonomy of Cultural Proxies
A central finding of the survey is the "definition void": none of the analyzed papers explicitly defined "culture." Instead, researchers rely on measurable stand-ins, or proxies. The authors propose a taxonomy dividing these into Demographic Proxies—such as Region, Language, and Religion—and Semantic Proxies—such as Values, Food, and Social Relations.
Geographical region and language remain the dominant proxies, used in 37 and 35 papers respectively. However, semantic coverage remains narrow; while there are many recognized semantic domains (like kinship or technology), the majority of studies focus almost exclusively on emotions and values.
Probing Methodologies
The survey also categorizes the methods used to "interrogate" these models. The vast majority of studies employ Black-box Probing, which evaluates text outputs through generative or discriminative prompts. In contrast, White-box Probing—which seeks to understand how culture is mechanistically encoded within model weights—is significantly underrepresented, with only one study in the survey attempting this approach.
Key Insights and Challenges
The review highlights several critical gaps in current research:
- Thin vs. Thick Descriptions: Borrowing from anthropology, the authors argue that NLP currently offers "thin" descriptions (observing surface-level outputs) rather than "thick" descriptions that capture the context and intent behind cultural behavior.
- The "Aboutness" Gap: While research often examines how a model speaks (e.g., sentiment or language), there is little focus on what it speaks about—the local news, history, and priorities that define a culture's "aboutness."
- Methodological Fragility: Cultural probing is highly sensitive to prompt wording; a model may appear culturally aware with one prompt and biased with a slightly rephrased version.
Future Directions
The authors conclude by calling for a more interdisciplinary approach, urging NLP researchers to collaborate with sociologists and anthropologists. Future work must move beyond simple translation-based multilingualism to capture unique cultural concepts and develop "situated" studies that evaluate how LLM biases impact real-world communities, particularly in the Global South.
Summary updated