Vision-language models (systems that answer questions about images) learn culture mostly from web text organized by country. Ask one about a photograph of a regional festival in Chhattisgarh, or a household ritual among Iran's religious minorities, and it tends to answer as if the image showed the country's most-publicized majority practice. This proposal treats that failure as a knowledge conflict: true, naturally occurring visual evidence pitted against the model's stored national-level stereotype. It asks a question existing work has not posed. When the model gets these cases wrong, does it fail to see the local practice, or does it see it and then override it while reasoning?
The answer decides what fix is worth building. If the local identity is represented inside the model and then suppressed, a training-free intervention on the internal activations should recover it, as similar interventions already do for colors and sizes. If the local identity is never represented, no amount of steering will help, and the field needs new perceptual data. We build a paired benchmark of authentic subcultural images with matched majoritarian "twins" and decoys. We trace where in a multi-step answer the majoritarian substitution happens. We then test whether existing steering methods transfer from physical counterfactuals to culture.
Core Claim
Our central hypothesis is that cultural proxy simplification (replacing fluid local culture with a national or majoritarian stand-in) shows up inside vision-language models as parametric stubbornness, and is mostly a reasoning-stage failure rather than a perception failure. Parametric means the answer comes from what is stored in the weights rather than from the image or prompt. Concretely, we predict three things. A linear probe on mid-layer activations will decode the local artifact's identity well above the rate at which the model names it. The majoritarian substitution will appear at a later step of the model's written reasoning, not at the recognition step. The gap will widen for under-documented, higher-poverty regions and for intangible practices.
Each prediction is falsifiable and grounded in the file. DOSA collected Indian subcultural artifacts through a community guessing game across 19 states. It reports text-only models swapping the regional festival Teej for the cinema-amplified Karwa Chauth, and it reports GPT-4 accuracy falling from 75% in Kerala to 22% in Chhattisgarh. Can I Introduce My Boyfriend to My Grandmother? classifies Iranian social norms. It finds models attributing majority Islamic rituals to non-Muslim citizens, and finds a GPT-4o F1 drop from 0.61 on general norms to 0.55 on Iran-specific ones.
On the conflict side, MMKC-Bench shows that multimodal models detect conflicts 75–79% of the time yet still answer from memory, and that larger models are more stubborn. That detect-but-ignore pattern is exactly the "sees but overrides" signature we expect for culture. All three papers motivate the proposal, but none puts images of local practices into a conflict setup.
What Is New Relative to Closest Work
The nearest pairwise work splits cleanly into two groups.
The first group adds culture to multimodal evaluation but has no conflict and stays at national granularity. GIMMICK evaluates 31 models on UNESCO heritage across 144 countries and 6 macro-regions. It reports GPT-4o at about 31.5% on Western image questions versus about 17% for Sub-Saharan Africa, and 8–10% on rituals versus about 30% on food. CulturalGround fine-tunes Pangea-7B on Wikidata entities from 42 countries and gains +5.0% on average, but explicitly excludes implicit norms and within-nation subcultures.
The second group builds conflict benchmarks with synthetic, culture-free edits. MMKC-Bench uses LLM-altered Wikipedia attributes and treats gestures as fixed meanings. Pixels Versus Priors builds purple lemons and oversized flies. Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning uses physical counterfacts such as six-toed feet. Both of the first two call for naturally occurring conflicts.
Our setting differs in kind, not just in topic. Both the local evidence and the national prior are true, at different granularities. The correct behavior is therefore not "always follow the image" or "always follow memory" but "follow the evidence that matches the scope of the question." That forces us to include decoys, so that a method which blindly trusts any local-looking signal is penalized. No prior conflict benchmark needs this.
Research Program
Benchmark construction. We will build a benchmark from DOSA's 615 artifacts and the Iran-specific portion of the ISN norms. We will start with a few hundred artifacts that have photographable form, prioritizing high-poverty states such as Chhattisgarh and the Iranian minority-religion contexts. Images will be sourced through community contribution in DOSA's participatory style rather than web scraping, since DOSA shows top-down sources under-document these artifacts. Each artifact is paired with a majoritarian twin: the national-level practice the model is most likely to substitute, chosen by querying text-only models DOSA-style and confirmed by community annotators.
Questions are three-step chains. The model must recognize the practice, infer the occasion or social role, and judge an appropriate behavior (for example, whether a given gift or action is expected, normal, or taboo, reusing the ISN label scheme). Each item appears under five conditions:
- image alone;
- image plus a retrieved national-level passage that asserts the majoritarian practice;
- image plus an accurate local passage;
- a decoy: a twin image with a false local caption;
- a decoy: a local image with a misleading majoritarian title, mirroring the misleading-title probes in Plot Twist.
Following MMKC-Bench's original-answer and counter-answer ratios, we score each response in two ways. The Majoritarian Answer Ratio is the fraction of items where the model gives the twin's answer. The Local Answer Ratio is the fraction where it gives the local answer. Scoring is per step of the chain, plus overall accuracy on decoys.
Seeing-versus-thinking diagnosis. For every failure we will run three tests.
First, we train linear probes on image-token and final-token activations, layer by layer, to classify local artifact versus twin. This reuses the probing approach of Tang et al., whose conflict probes reach 93.2–98.8% AUC concentrated in layers 15–22 of 7B models.
Second, we apply a logit-lens readout to locate any "flip" between prior and evidence. This follows Pixels Versus Priors, whose logit-lens analysis finds such a flip in the middle layers.
Third, we run a verbal-recognition test in the style of Bad Seeing or Bad Thinking? (MoCA). The model writes what it recognizes, and a text-only model answers from that description alone.
The contrast between the probe and the verbal test is itself informative. MoCA notes that its text bottleneck breaks for non-symbolic content. If probes decode local identity for intangible rituals where verbal recognition fails, verbal verification is the wrong tool for cultural perception.
Intervention. We will compute two steering vectors (directions added to hidden activations at inference time). One is a pixels-versus-priors vector from contrasting prompts, as in Pixels Versus Priors. The other is a context-faithfulness vector built from about 1.5k examples, as in ContextFocus. We apply each uniformly, and also gated by the conflict probe so that it fires only when the probe detects an effective local-versus-national conflict. We will also test head-level dampening in the style of PH3, which finds separate memory heads and context heads in text models. This checks whether those heads exist and carry the cultural conflict in multimodal models.
Baselines
The naive reference is the unmodified model answering from the image alone, and under the national-passage condition. We use Qwen2.5-VL at 3B, 7B and 72B, matching MMKC-Bench's scaling comparison, plus LLaVA-Next, matching Pixels Versus Priors. This anchors how bad the problem is and whether scale worsens it. MMKC-Bench predicts the 72B model will be more stubborn.
The strongest prompting competitor is geographic hinting. GIMMICK reports that adding country or region names significantly boosts accuracy, especially for under-represented regions. We run it at the national level and also at the state or community level, the stronger version a reviewer would demand. A state-level hint is a cheap fix that may simply work, so we treat it as the main threat.
The strongest training-free internal competitor is uniform pixels-versus-priors steering. Its paper reports shifting 99.3% of color predictions and 80.8% of size predictions from prior to visual evidence. Those are the only reported numbers for this family of methods, and they come from physical counterfacts. For subcultural conflicts we found no reported result on Lacuna for pixels-versus-priors steering, ContextFocus, or any other method. Both steering vectors will be fit on the same held-out calibration split, with the same layer-and-strength search budget, as our gated variant.
The strongest trained competitor is CulturalPangea against its base Pangea-7B. This tests whether entity-level cultural fine-tuning already transfers to sub-national practices. The reported reference is +5.0% on average across CVQA, MaRVL, ALMBench and MERLIN.
We also run Chain-of-Note style reading notes from Zhang et al., which judge each passage's relevance and trustworthiness before answering.
The ablation that isolates our key ingredient compares probe-gated steering with uniform steering at matched strength. Beating the baselines means three things together. The gated method must raise Local Answer Ratio on authentic local items in conditions (1) and (2). It must not lower decoy accuracy in conditions (4) and (5). It must not lose accuracy on GIMMICK CIVQA and CVQA, which serve as collateral-damage controls. The comparison is against the best of hinting, uniform steering and CulturalPangea under equal tuning budgets.
Why not just…?
Why not just give the model a precise location hint? This is a real threat, and GIMMICK's hint results suggest it will help. The Iranian norms paper offers counter-evidence: an explicit "Iran prompting" condition did not stop models from over-generalizing majority rituals to minorities. A hint names the region but does not name the community, and the failure we target is within-region. Our decoy conditions settle the rest. If state-level hints raise local accuracy without raising acceptance of false local captions, hints are the right answer, and we will say so.
Why not just apply an existing steering vector toward the image? Tang et al. report that steering toward visual evidence is easier than steering toward text, so uniform pixels-versus-priors steering may transfer well. The risk is collateral damage. Pushing every answer toward the pixels should hurt non-conflict items and cases where the image is ambiguous and the text is the reliable cue. The gated-versus-uniform ablation measures exactly this. If uniform steering matches the gated version on conflict, decoy and control sets, we drop the gate and report that off-the-shelf steering transfers to culture. That would still be a useful, cheap result.
Why not just retrieve local text or fine-tune on more cultural data? DOSA shows that top-down sources lack documentation of these artifacts, so there is often nothing to retrieve. SCoRe, a structured-context method for knowledge-based visual question answering, notes that models mis-arbitrate when retrieved passages contradict memory. Our condition (3) tests directly whether an accurate local passage overcomes stubbornness. On fine-tuning, GIMMICK reports that attaching vision encoders can erase cultural knowledge present in the text backbone. The CulturalPangea comparison tells us whether entity-level fine-tuning reaches sub-national practices.
Decisive Experiments
The first decisive check is the seeing-versus-thinking fork, run on the image-alone and national-passage conditions. The thinking-failure hypothesis predicts three findings. Probes decode the local identity from mid-layer activations at a rate clearly exceeding the Local Answer Ratio. The written chain names the local practice at the recognition step before substituting the twin later. Probe-gated steering recovers a substantial share of the lost answers. If instead probes sit near chance on the very items the model gets wrong, the hypothesis is refuted. The project then pivots to perception data, and steering is shown not to transfer from colors to culture.
The second check is the stratified analysis. We split the probe-minus-output gap by DOSA's poverty gradient, by tangible versus intangible practice (following GIMMICK's split), and by model scale. We predict the gap grows with marginalization and with scale. That would tie cultural proxy simplification to the scale-induced stubbornness reported in MMKC-Bench.
The third check compares probe and verbal recognition on intangible rituals. It tests whether verbal perception checks like MoCA's systematically under-credit cultural perception. If so, any perception-rewarding training for cultural reasoning should verify perception internally rather than through text.
References
- DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures
- Can I Introduce My Boyfriend to My Grandmother? Evaluating Large Language Models Capabilities on Iranian Social Norm Classification
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
- GIMMICK: Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking
- Grounding Multilingual Multimodal LLMs With Cultural Knowledge
- Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts
- Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
- Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning