Culturally Grounded Vision Language Models

This direction is merged into another direction: cultural representation gap. Its last version stays readable and citable here.

Version 1 of 132 papers32 conceptsfrozen 2026-10-04

Most Vision-Language Models (VLMs) see the world through a Western lens. Because they are trained on massive datasets scraped from the English-speaking internet, they inherit a "Western-centric" bias that limits their ability to understand the nuances of global traditions, daily life, and regional artifacts. Recent research aims to move beyond simple translation toward cultural grounding—ensuring that AI can reason about visual information within the specific social and aesthetic frameworks of diverse global populations.

The Western Lens and the Cultural Gap

The fundamental problem is a representational gap. As Fangyu Liu, Emanuele Bugliarello, and colleagues demonstrated with the MaRVL benchmark, models that excel at English-centric tasks often fail when faced with concepts and images from the non-Western world. This isn't just a language problem; it's a visual reasoning problem.

Models often rely on "cultural shortcuts." In the MIXCUBE study, Jun Seong Kim and Alice Oh found that models often identify the "culture" of an object based on the ethnicity of the person holding it rather than the object itself. If a model sees a person of one ethnicity eating food from another culture, its performance can drop by over 50%. This suggests that models are not truly "seeing" the cultural artifact, but are instead relying on superficial demographic correlations.

Beyond Translation to Functional Mapping

A common pitfall in making AI global is simply translating English datasets into other languages. However, Jean de Dieu Nyandwi and Yueqi Song argue in their work on CulturalGround that translation fails to address the underlying Western-centric content. If the images themselves only show Western kitchens, translating the word "stove" doesn't help the model recognize a traditional cooking hearth.

One powerful intuition for solving this is shifting from object-centric to function-centric labeling. In the Culture Affordance Atlas, Joan Nwatu and Rada Mihalcea propose categorizing objects by the universal human functions they fulfill. Instead of just looking for a "bed," the model learns to identify objects used for "sleeping." This allows the model to reconcile diversity—recognizing that a floor mat in one culture and a raised mattress in another serve the same human purpose, thereby reducing the performance gap between high- and low-income data.

Language as a Perceptual Lens

The bias is rooted deeper than just the images; it starts with the text. Amith Ananthram and Mohit Bansal showed in See It from My Perspective that the text-only pre-training of the underlying Large Language Model (LLM) sets a cultural baseline. Simply prompting a model in a different language at the end isn't enough if the model wasn't exposed to that culture's linguistic "lens" during its initial training.

Linguistic backgrounds influence what people prioritize in a scene. For example, some cultures focus on central objects, while others prioritize the broader context. Models trained on English-only data often adopt a "universalist" assumption that ignores these variations in human perception.

Benchmarking Global Nuance

To track progress, researchers have developed massive-scale, culturally specific benchmarks:

  • Food and Cuisine: Genta Indra Winata and Yutong Wang introduced WorldCuisines, a dataset of 1 million points across 30 languages, revealing that models struggle with regional ingredients and adversarial cultural contexts.
  • Regional Specificity: Benchmarks like Rice-VL (Southeast Asia), TaiwanVQA, and K-Viscuit (Korea) test models on local markets, religious symbols, and daily social norms.
  • Scaling Data: Projects like DEJIMA (Japan) and SEA-VL (Southeast Asia) use automated pipelines to crawl and ground millions of image-text pairs, ensuring that models see local scenery and customs that are absent from Western-centric datasets.

The goal of this research direction is to move from models that are "multilingual" (can speak many languages) to models that are "multicultural" (can understand the world through many different eyes). By grounding vision-language learning in structured global knowledge, as seen in the CulturalPangea model, AI can begin to overcome its Western blind spots.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_culturally_grounded_vision_language_mode_3d7351c1,
  title = {Culturally Grounded Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/culturally-grounded-vision-language-models/rdr_3d7351c13ea64d22b9001838cdd1f96d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_culturally_grounded_vision_language_mode_3d7351c1_v1,
  title = {Culturally Grounded Vision Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/culturally-grounded-vision-language-models/rdr_3d7351c13ea64d22b9001838cdd1f96d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}