Seeing or Thinking in the Native Language? A Budgeted Test of the Multilingual Multimodal Gap

Vision-language models answer questions about a culturally specific image noticeably worse when the question is asked in that culture's own language than when it is asked in English. On one multilingual cultural benchmark, LLaVA-1.5-7B drops from 49.6% to 35.5% (CVQA). Nobody knows whether this "native language penalty" comes from the model failing to see, meaning it misreads non-Latin scene text or fails to recognize local objects, or from the model failing to think, meaning it cannot carry out the deduction once the question is in a non-English language. We propose to separate the two by experiment. Native-speaker annotators write a plain-text description of what the image shows that is relevant to the question, without answering it. We call this a "perception anchor." We then replace the image with this human description and query a text-only "blindfolded" solver in both languages. If the penalty survives with perception held fixed by a human, it is a thinking failure. Whatever disappears was a seeing failure.

The catch is that perception anchors are expensive human annotations, and different annotators describe the same image differently. That variation is partly cultural: a local annotator names the dish or reads the sign, while an outsider describes shapes and colors. The second half of the proposal treats that disagreement as a variance component to measure and budget for, not as noise to average away. We show how many anchors per item, and from whom, a fixed budget should buy. This directly tests whether the "one label per item" rule of Don't Label Twice carries over to this kind of measurement.

Core Claim

The native language penalty can be split into a perception part and a reasoning part by intervening on perception with human-written anchors. The split depends on who writes the anchor, and for culturally grounded items that dependence is large enough that the budget-optimal design collects more than one anchor per item. We expect the split to differ sharply by benchmark type. On text-in-image questions in non-Latin scripts, seeing should dominate. On native-language exam reasoning, thinking should dominate. That pattern would reconcile two claims in the literature that currently look contradictory.

Let $A(\ell, v)$ be a model's accuracy when the question is in language $\ell \in \{\text{nat}, \text{en}\}$ and the visual input $v$ is either the image ($\text{img}$) or a human perception anchor ($\text{anc}$). For the anchor condition, the native anchor is professionally translated to English. Text-only translation does not create the image–text mismatch that MTVQA documents for translated visual benchmarks. The decomposition is

$$ \underbrace{A(\text{en},\text{img}) - A(\text{nat},\text{img})}_{\text{penalty } P} = \underbrace{A(\text{en},\text{anc}) - A(\text{nat},\text{anc})}_{\text{thinking } T} + \underbrace{\big(P - T\big)}_{\text{seeing } S}. $$

Here $T$ is the language gap that remains when perception is supplied by a human. $S$ is the part of the penalty that only exists when the model must look at the image itself.

Research Program

The motivating evidence pulls in two directions. MTVQA, a nine-language text-in-image benchmark, attributes 39% of failures to text recognition (OCR) and 34% to reasoning, and finds steep drops on Arabic and Russian relative to French. GAOKAO-MM, built from Chinese college entrance exam questions, reports that models identify diagram parts but fail the multi-step logic. GeoSQA goes further: models still fail on Chinese geography scenarios when diagrams are replaced by human text. Both kinds of evidence are post-hoc error labeling or single-language studies. Neither crosses language with perception in one controlled design.

The method borrows from Bad Seeing or Bad Thinking?. That paper has a model write its perception into a recognition block and asks a text-only proxy (Qwen2.5-14B) whether that block suffices to solve the problem. It uses this signal for credit assignment in English reinforcement learning. We keep the blindfolded-proxy idea but make three changes. We use it for diagnosis rather than training. The anchors come from humans, so model perception is not graded by model perception. And we run the proxy in both languages.

The same paper warns that the proxy can fail even when the perception is correct. We control for this by reporting $T$ only on items where the English solver succeeds with the English anchor. We also run two solvers: the VLM's own language backbone with the image withheld, and a separate multilingual text model.

Anchor collection follows a strict rubric. Annotators describe visible content, transcribe scene text in its original script, and name recognizable entities. They must not state or imply the answer. Every anchor goes through a leakage filter: a solver given the anchor and the answer options but no question must not beat chance. Anchors that fail are rewritten.

For each item we collect two anchors from native-speaker annotators of that culture and one from a fluent non-native annotator who lacks local cultural familiarity. This design treats annotator background as a factor, the role that the subjectivity literature assigns to disagreement. Training Affective Computer Vision Models by Crowdsourcing Soft-Target Labels shows that collapsing many raters to one hard label hides real perceptual variation, but it paid for 100 ratings per image. We want the minimum depth that preserves the signal.

The budget analysis treats each anchor as a draw within an item. The item-level contrast $T_i$ has between-item variance $\sigma_b^2$ and between-annotator variance $\sigma_w^2$. With $N$ items, $m$ anchors per item, per-item fixed cost $c_0$ (item curation plus model runs) and per-anchor cost $c_a$, standard two-stage sampling gives

$$ \mathrm{Var}(\hat T) = \frac{\sigma_b^2}{N} + \frac{\sigma_w^2}{Nm}, \qquad m^\star = \sqrt{\frac{\sigma_w^2\, c_0}{\sigma_b^2\, c_a}}. $$

The optimal $m^\star$ comes from minimizing that variance subject to $N(c_0 + m c_a) = B$, where $B$ is the total budget. The pilot estimates $\sigma_w^2$ and $\sigma_b^2$ separately for each language and each benchmark.

This differs from Don't Label Twice in a specific way. That paper proves single labels are optimal for ranking binary classifiers when labels are noisy views of an objective truth, and says itself that the result does not cover inherent ambiguity. Our estimand is a perception-conditioned accuracy gap, and our $\sigma_w^2$ is partly genuine cultural difference in what is seen. We predict $m^\star > 1$ on culturally grounded CVQA items and $m^\star \approx 1$ on MTVQA text-transcription items, where anchors should mostly agree.

Baselines

The naive reference point is the undecomposed penalty $P$ as CVQA reports it, for example the 49.6% to 35.5% English-to-local drop for LLaVA-1.5-7B and 75.4% versus 74.3% for GPT-4o (CVQA). It tells us the size of the gap but not its cause. We recompute it on our subsets with the same prompts and decoding settings for every condition.

The strongest existing diagnostic is MTVQA's manual error taxonomy (39% OCR, 34% reasoning, 15% language bias, 12% hallucination; MTVQA). We replicate it on our items by having two annotators label each failure post hoc, at the same annotation budget we spend on anchors. To beat it, our split must predict held-out intervention effects better than the taxonomy's proportions do (see the decisive check below).

The strongest existing intervention-style baseline is the OCR pipeline. MTVQA reports that OCR followed by GPT-4V reaches 28.3%, versus 22.0% for end-to-end GPT-4V. Feeding OCR output instead of human anchors gives a cheap automatic perception intervention. We run it with the same solvers and the same items.

The model-self-perception baseline follows Bad Seeing or Bad Thinking?. The VLM writes its own recognition block in each language, and the blindfolded solver uses that block as the anchor. This costs nothing in human labor.

Two ablations isolate the key ingredients. The first uses one native anchor per item and spends the saved budget on more items, which is the Don't Label Twice design. The second swaps native anchors for non-native ones, which isolates how much of $S$ is cultural recognition rather than script reading.

Why not just…?

Why not just let the model write its own perception description, as MoCA does? It is free, and if it agreed with human anchors, our human budget would be wasted. The problem is circularity: the model's native-language perception is exactly what the penalty may be measuring, so using it as the anchor can hide a seeing failure inside $T$. We test this directly rather than assuming it. If self-anchors and human anchors give the same $S$ and $T$ within confidence intervals on a benchmark, we will recommend self-anchors for that benchmark and report that human anchors were unnecessary there.

Why not just label each failure as "OCR" or "reasoning," as MTVQA did? That approach exists and is cheaper per item. But post-hoc attribution is itself an annotator judgment, and it is exactly the step on which MTVQA and GAOKAO-MM reach opposite pictures. Our design replaces judgment with intervention. The comparison is whether either method's split predicts what happens when you supply oracle perception or translate the question, on held-out items.

Why not just translate the native question into English and compare (translate-test)? CVQA's English-versus-local contrast is essentially this, and it measures only $P$. A model can do better in English because the English question carries recognizable text cues, the shortcut Plot Twist shows inflates chart accuracy, and not because it reasons better in English. Our anchor conditions hold the visual content fixed while varying language, which translate-test cannot do.

Why not just collect one anchor per item and maximize coverage, per Don't Label Twice? This is a real threat. If $\sigma_w^2$ turns out small, it is the right answer and we will say so. The pilot estimates the variance components, and the single-anchor ablation reports the achieved confidence-interval width at equal cost. We stand by the multi-anchor design only where it gives tighter intervals or reveals a native-versus-non-native difference that a single anchor would miss.

Proposed Experiments

We draw about 300 items per language from CVQA for six languages spanning Latin and non-Latin scripts, about 300 per language from MTVQA for Arabic, Russian, Thai, and Japanese, and a GAOKAO-MM subset in Chinese. GAOKAO-MM has no native/English contrast by default, so its questions are professionally translated. Its diagrams have little embedded text, which limits the image–text mismatch problem. For reference, GAOKAO-MM reports GPT-4V at 48.1% and all open-source models below 25% (GAOKAO-MM). We evaluate GPT-4o, an InternVL model via the InternVL codebase, a multilingually instruction-tuned model via Parrot, and LLaVA-1.5-7B. All model and benchmark choices come from the supplied material.

We report $P$, $T$, and $S$ with bootstrap intervals that resample both items and annotators, along with $\hat\sigma_b^2$, $\hat\sigma_w^2$, and $m^\star$ for each benchmark and language. The first prediction is that on MTVQA non-Latin languages $S$ accounts for most of $P$. The second is that on GAOKAO-MM $T$ dominates. The third is that on CVQA, non-native anchors yield a larger $S$ than native anchors, meaning outsider descriptions lose recognition the model also lacks. If CVQA's $S$ does not depend on who wrote the anchor, the subjectivity component of the proposal fails and we drop it.

The decisive check is predictive validity. On held-out items we apply two real interventions. One supplies perception (OCR or anchor-augmented input). The other changes the reasoning language (translate the question to English while keeping the image). The decomposition passes if, for each model and benchmark, the measured $S$ and $T$ predict which intervention closes more of the penalty, and does so better than the post-hoc taxonomy proportions. If neither method predicts intervention outcomes, the seeing/thinking dichotomy is the wrong frame for this penalty, and that negative result is worth reporting.