Locating the Native-Language Penalty Inside the Image

Vision-language models answer questions about the same photograph much worse when the question is asked in the photographer's own language than when it is asked in English. On CVQA, a benchmark of culturally grounded photos with parallel native-language and English questions, LLaVA-1.5-7B scores 49.6% in English but 35.5% in the local language (CVQA). The image and the question's meaning are held fixed, so the only thing that changes is the language of the question. This note proposes to find out which parts of the image absorb that penalty. The tool is visual topic modeling, a family of unsupervised methods that cut images into small patches, map each patch to a discrete "visual word," and learn recurring co-occurrence themes ("topics"). Each patch then gets a topic, and each image gets a mixture of topics.

The idea fits in two sentences. Build a topic model directly on the vision encoder's own patch features, so every image in a multilingual benchmark becomes a mixture of topics and every patch has a topic label. Then test whether the English-versus-native accuracy gap, and the shift in where the model looks when the question language changes, concentrate on particular visual topics rather than spreading evenly. If they do, the answer is a finer map of cross-lingual failure than human domain labels provide. It also gives a practical lever: correct the model only where the problematic topics appear.

Core Claim

The claim is that the native-language penalty is visually localized. It is carried by a small number of patch-level visual topics, and these topics predict the per-question gap better than either the benchmark's hand-assigned domain categories or clusters of whole-image embeddings. When the question switches from English to the native language, the model's attention over image patches moves away mostly from patches belonging to those high-gap topics.

The within-image, patch-level half of this claim is what is new. CLAIM showed that non-English queries make intermediate cross-modal attention heads (layers 10–25) drift off the relevant image regions. CLAIM measured this only for yes/no object-presence probes, and it treated the drift as a single language-wide shift. CVQA and All Languages Matter (ALM-bench) measured where the penalty falls, but only across coarse human categories such as "Customs" or "Notable Key Figures." Neither can say which part of an image is responsible. Neither can handle images that mix, for example, generic street scenery with ceremonial dress.

If the claim holds, it suggests a direct intervention: estimate CLAIM-style correction vectors separately for each topic, and weight them by the image's own topic mixture. This is instead of using one correction per language.

Research Program

The first component is the topic model. We freeze the vision encoder of the model under study, such as LLaVA-1.5-7B's CLIP encoder. We run k-means on its patch tokens, pooled from a large image set disjoint from the evaluation benchmarks, to form a codebook of visual words. We then fit LDA with each image's patches as one document.

Classical visual topic models used SIFT codebooks for this step (Monay et al.; Hörster and Lienhart). Using the encoder's own features instead means the topics describe what the model actually sees, not what an external descriptor sees. The classical literature is useful here in a specific way. Monay et al. showed that conditioning patch interpretation on the image's own topic mixture beat dataset-wide priors: 72.4% versus 68.5% half total recognition rate on their Corel subset. We therefore assign patch topics from the per-image posterior, not by nearest codeword. Russell et al. showed that coherent objects form tight topic clusters. We use that property as a sanity check that the topics are meaningful.

Codebook granularity is a known weak point of these methods, so we sweep codebook size and topic count $K$ and report whether the conclusions hold across the sweep.

The second component is the gap model. For each CVQA question $i$, let $g_i \in \{-1,0,1\}$ be English correctness minus native correctness. We fit a mixed-effects model of the form

$$ \Pr(\text{native wrong},\ \text{English right} \mid i) = \sigma\big(\beta^\top \theta_i + \gamma^\top z_i + u_{\text{lang}(i)}\big), $$

where the terms are defined as follows. $\theta_i$ is the image's topic mixture. $z_i$ collects text-side controls: tokenizer fertility of the native question, meaning subword tokens per word; question length; script; and the benchmark's domain label. $u_{\text{lang}(i)}$ is a per-language random intercept, and $\sigma$ is the logistic function. The quantity of interest is how much held-out predictive likelihood $\theta_i$ adds on top of $z_i$ and the language intercept. Cross-validation splits are made by image, so no photograph appears in both training and test folds.

The third component is the attention analysis. CVQA supplies the same image with parallel questions, so we can compare the model's attention over image patches under the English and native prompts. We measure the divergence between the two attention distributions, restricted to the intermediate layers that CLAIM identified. We then break that divergence down by patch topic. The prediction is that the topics with large $\beta$ coefficients also account for a disproportionate share of the attention shift.

This speaks to the tension raised by Frank et al.. They found that English multimodal transformers barely use text when processing visual regions, which seems to conflict with CLAIM's finding that the question language does move visual attention. A topic-localized drift would reconcile the two: language could leave most visual processing untouched and still disrupt specific themes.

The fourth component is the intervention. Following CLAIM, we identify language-sensitive attention heads and estimate shift vectors from a calibration set of English/native pairs. The difference is that we estimate one shift vector per dominant topic. At inference, we apply a mixture of these vectors weighted by the test image's $\theta$.

Baselines

The naive reference point is a language-only gap model, meaning the per-language intercept alone. It says the penalty is a constant tax per language and predicts every item in that language identically. Any visual account must improve on this held-out likelihood by a margin larger than the variation across cross-validation folds.

The strongest competing explanation is the benchmark's own human domain labels: CVQA's 10 categories, plus ALM-bench's 19 domains when we repeat the analysis there. They are strong because native experts assigned them and they already carry cultural meaning. Both feature sets enter the same mixed model under the same cross-validation. "Beating" this baseline means the topic mixture adds significant predictive likelihood beyond the domain labels, measured by a likelihood-ratio test on held-out folds. A bare correlation with the gap would not be enough.

The ablation that isolates the patch-level ingredient is k-means clustering of whole-image CLIP embeddings, with the number of clusters equal to $K$. It uses the same encoder but gives each image a single cluster, with no within-image structure. If this baseline predicts the gap as well as topics do, the mixture and patch structure add nothing to prediction. The attention analysis would then remain the only thing topics can do that it cannot.

For the intervention, the strongest existing method is CLAIM's global per-language shift. It reports an average POPE accuracy gain of 13.56% across 7 languages, up to 30% in Spanish, and 21.75% on MME hallucination subsets. We found no reported result on Lacuna for CLAIM applied to CVQA or ALM-bench, so we rerun it ourselves on LLaVA-1.5-7B. It uses the same calibration pairs and head-selection budget as our method.

A second ablation controls for the extra correction vectors. It uses per-group shift vectors estimated from a random partition of the calibration items, matching our number of vectors and calibration data. If topic-conditioned shifts beat this random-group control and global CLAIM, then the topics, and not just the added capacity, explain the gain.

We report Parrot as a trained reference, not a like-for-like competitor. It routes visual tokens through language-specific expert networks and reports 79.94% on MMMB Turkish against Qwen2-VL's 79.33%.

Why not just…?

Why not just slice the gap by the benchmark's domain labels? This is the most serious threat. Domain labels already say that "Notable Key Figures" is harder than "Education and Heritage" for GPT-4o (72.7% vs 83.7% on ALM-bench). If domain labels explain the item-level gap as well as topics do, we will report that and drop the prediction claim. Two things remain regardless of that outcome. First, domain labels are one per question, while images mix themes. Second, labels cannot localize attention shift to regions of an image. The nested likelihood test settles the first issue, and the attention decomposition addresses the second.

Why not just cluster whole-image embeddings? It is cheaper and uses the same encoder. The same-size k-means ablation measures directly whether patch-level mixtures add predictive value. If they do not, the case for topic modeling rests only on the within-image attention result, and we will say so.

Why not assume the gap is purely a text problem, from tokenization or poor question comprehension? Tokenizer fertility drives cross-lingual variation in text-only settings (Team DMG), and it may explain much of the CVQA gap. That is why fertility, length, and script are controls in $z_i$. If topics add nothing beyond them, the penalty is not visually localized, and that is a clean negative result. CLAIM's attention evidence suggests perception is also involved, but our analysis does not presuppose it.

Why not just translate the question to English? For CVQA, the English condition is effectively translate-test with a human translator, so it is the realistic upper bound for this route. We also run an automatic MT pipeline, because Parrot reports a seesaw effect: machine translation helped Chinese but hurt Russian and Portuguese. Translate-test is a practical fallback, not a counterargument to the diagnostic question. Our intervention is judged by how much of the English–native gap it recovers without any translation.

Proposed Experiments

The primary testbed is CVQA, chosen for its parallel native/English questions over identical images across roughly 30 languages. The primary model is LLaVA-1.5-7B, chosen because its reported 49.6/35.5 split gives a large and documented gap and because its attention is accessible. Qwen2-VL serves as a second white-box model. Replication uses ALM-bench's 100 languages and 19 domains to test whether high-gap topics recur across benchmarks and scripts. Here the English reference must come from translated prompts, because ALM-bench does not provide CVQA's native/English pairing.

The first decisive check is predictive. Across the $K$ and codebook sweep, does the topic mixture significantly improve held-out gap likelihood over language intercepts plus domain labels plus text controls, and over whole-image clustering? A positive result requires this to hold at most settings, not a single chosen $K$.

The second decisive check is spatial. Is the share of English-versus-native attention divergence falling on high-gap-topic patches larger than their share of image area? We compare against a permutation null that shuffles topic labels across patches within each image.

The third check is causal and practical. On CVQA's native-language split, does topic-conditioned shifting recover more of the English–native gap than global CLAIM and the random-group control? It must not reduce English accuracy. We also report POPE in CLAIM's 7 languages, so the comparison includes CLAIM's own benchmark.

As a guard against CLAIM's acknowledged risk of pushing interpretation toward Western perception, we report gains separately for high-gap cultural topics and generic topics. This shows whether corrections toward English attention patterns help or hurt the culturally specific content.

The outcome is informative either way. A localized penalty would give a visual target for data collection and targeted correction. A null result, with a uniform or purely text-driven penalty, would redirect effort to tokenization and language-side alignment.