Open vision-language models, meaning models that answer questions about images, answer worse when the same question about the same image is asked in the user's own language rather than in English. On the culturally diverse CVQA benchmark, LLaVA-1.5-7B drops from 49.6% accuracy in English to 35.5% in the local language (CVQA). The image has not changed, so the lost accuracy is not a vision failure. We propose to test a specific explanation and a cheap fix. The explanation is that the model often fails to link the handful of content words in a low-resource question (a dish, a garment, a spatial relation, a kinship term) to concepts it already knows. The fix is to look those words up in standardized cross-linguistic lexicons, meaning public dictionaries and wordlists stored in shared formats with language and concept identifiers, and to insert short glosses next to the original question. The question itself is left untranslated.
This is retrieval-augmented generation, where the model receives looked-up text at answer time, with a twist. The retrieval is an exact lookup keyed on standardized identifiers, not a similarity search in an embedding space. That matters because dense retrievers are weakest in exactly these languages (VisR-Bench). The data already exists but is scattered: PanLex covers 13,213 languages, Wikidata 10,287, and Glottolog 8,604 (van Esch et al.). Linguists have also built the tooling to pull such data into one tabular standard, Cross-Linguistic Data Formats (CLDF) (Forkel et al.). No one has asked whether this infrastructure can repair multimodal question answering at inference time.
Core Claim
A large share of the native-language penalty in open vision-language models is a lexical grounding failure. It can be recovered without translation and without fine-tuning. The method has three steps. First, map each content word in a local-language question to a concept identifier through a merged CLDF lexicon. Second, attach the English label and a short description for that identifier. Third, keep the original wording, so culture-specific terms are never forced through a machine translation system that may flatten them.
The claim is falsifiable in two directions. If glossing every content word still leaves most of the gap, the penalty lives in reasoning carried out in the weak language, not in vocabulary. That is itself a useful finding. It would extend the perception–reasoning split documented for math diagrams (MATH-Vision; GeoSQA) to a language–reasoning split. If ordinary machine translation of the question recovers as much as glossing does on low-resource and culture-heavy items, the standardized lexicon adds nothing, and we will say so.
What Exists and What Does Not
Three kinds of prior work matter here: work that motivates the proposal, work that constrains it, and work that supplies its methods.
The motivating evidence comes from benchmarks that diagnose the gap without repairing it. CVQA measures the native-language penalty across 31 languages and finds near-chance accuracy for languages such as Breton and Mongolian. IndicVisionBench finds sharp degradation in lower-resource Indic scripts such as Odia. xGQA translates the GQA test set into 7 languages and shows a zero-shot drop from about 60% to about 20%. It also finds that 48 translated examples recover up to 20 points. That result suggests the missing ingredient is small and lexical rather than a deep capability. None of these benchmarks tries inference-time retrieval.
The existing fixes are parametric. PALO fine-tunes on 2.1M machine-translated instructions and gains 32.3 points on Multilingual LLaVA-Bench for Hindi, Arabic, Bengali and Urdu. VisCPM/MPM pivots through English on the assumption that visual concepts are language-agnostic. Both require training, and both inherit English framing, which the cultural benchmarks show breaks on region-specific concepts. Our method trains nothing and uses concept identifiers rather than English sentences as the shared pivot. This follows the pivot-format argument of Ide and Romary: converting $N$ resources through one shared representation costs linear rather than quadratic effort.
Two results constrain the design. VisR-Bench shows that multimodal retrievers pay a "multilingual tax": text retrievers such as BGE-M3 beat them in Finnish, Czech and Arabic, and all models do worst on tables. This argues against embedding-based retrieval in weak languages and in favor of keyed lookup. Pivot Through English retrieves matching English questions and beats translate-test by 2.7% on XQuAD and 6.2% on MKQA. Its authors concede that it fails on local entities absent from English resources. Our lookup targets exactly those entities, because lexicons such as PanLex and Wikidata labels hold native terms that English question banks do not.
Two lines of work supply methods. DCoT shows that inserting retrieved text at inference time helps vision-language models without training, and that retrieval on question text alone beats image-plus-text retrieval. We adopt the text-side, training-free design but retrieve lexical facts instead of 80 fixed English demonstrations. The CLDF conversion tooling of Forkel et al. is the ingestion method. Work on fieldwork data (Tagging Ingush; SHR++) shows that interlinear glosses and structured morphological analyses support usable analyzers for inflected forms in otherwise data-poor languages. We need such analyzers because dictionary headwords rarely match the forms that appear in questions.
Research Program
We will build a merged lexical grounding table in CLDF. It will contain PanLex expressions, Wikidata multilingual labels and descriptions, and, where available, CLDF wordlists and fieldwork lexicons. Every row will be keyed by a Glottolog language code and a concept identifier (a Wikidata item or a meaning class). Building the table is itself a contribution: it turns the scattered aggregators surveyed by van Esch et al. into something a model pipeline can query.
At inference time, the question and answer options are segmented. Content words are normalized with the best available analyzer for the language (a CLDF-derived form list, a fieldwork-trained analyzer, or a character-overlap fallback) and looked up by exact key. Hits are rendered as compact inline annotations, for example the original term followed by its concept label and a one-line description. The annotated question then goes to an unchanged vision-language model.
We will log coverage for every item: the fraction of content tokens that received a gloss. Coverage is the mechanism variable. If the lexical account is right, the per-item accuracy gain should rise with coverage, and items with zero coverage should show no gain.
Baselines
The naive reference is the unmodified model answering the local-language question, reported alongside the same model answering the parallel English question. The English score is the ceiling. xGQA provides exact English parallels, and CVQA reports both conditions. The published anchors are 49.6% English versus 35.5% local for LLaVA-1.5-7B on CVQA, 75.4% versus 74.3% for GPT-4o on CVQA (CVQA), and roughly 60% versus 20% zero-shot on xGQA (xGQA). Our headline metric is the fraction of this English–local gap that each method recovers.
The strongest competitor is translate-test: the question and options are translated into English with a strong machine translation system, then answered by the same model. It uses the same inference-time, training-free budget and attacks the same failure. We will run it with the best available translation model for each language and the same prompt template, and report it on every item where our method is run. No reported result was found on Lacuna for translate-test on CVQA or xGQA, so we measure it ourselves. The related text-QA result, where RM-MIPS beats translate-test by 2.7% on XQuAD and 6.2% on MKQA (Pivot Through English), shows that translate-test is beatable but not weak. We count it as beaten only if glossing is significantly better on the low-resource and culture-specific subsets, using paired bootstrap tests over items. Near parity on high-resource languages is acceptable.
The strongest parametric competitor is PALO-13B (PALO), evaluated on its supported languages. No reported result was found on Lacuna for PALO on CVQA or xGQA. We will also apply glossing on top of PALO to test whether the two gains add up.
A dense-retrieval baseline indexes the same lexicon rows with BGE-M3 and retrieves the top entries by similarity to the question. The strongest text retriever in the low-resource VisR-Bench languages is BGE-M3 (VisR-Bench), which makes it the fair choice. This baseline isolates our key ingredient: same content, keyed versus embedding-based access.
Two control ablations complete the set. The first inserts glosses of the same length for random concepts, which tests whether gains come from information rather than prompt length or format. The second is an "English label only" variant that drops descriptions, which tests how much the gloss needs beyond a word-to-word mapping.
Why not just…?
Why not just translate the question? This is the real threat, and the experiments are designed around it. Translation is expected to win or tie on xGQA, whose questions are themselves translations of English and contain few culture-bound terms. The case for glossing rests on two claims we can check. First, translation quality collapses for languages such as Breton, Mongolian or Odia. Second, translation mistranslates or flattens local entities. The evidence that machine-generated content is unreliable here is indirect: IndicVisionBench reports up to 62.5% edit rates on model-drafted Indic questions, and PALO needed human-corrected seeds to avoid translationese. We will tag CVQA items for whether the answer hinges on a named cultural entity. If translate-test matches glossing on those items too, the lexicon offers no advantage and we will report that. A combination condition (translation plus glosses) tests whether the two methods are complementary.
Why not just use a frontier model? GPT-4o shows almost no penalty on CVQA (75.4% versus 74.3%), so for closed frontier models the problem is largely solved. The proposal targets open and small models, which communities working in low-resource languages can actually deploy and which lose more than 14 points. We will state this scope plainly rather than claim a universal fix.
Why not just retrieve native-language Wikipedia passages? Passage retrieval in weak languages runs into the multilingual tax that VisR-Bench documents. Many target languages also have small or empty Wikipedias. We include the dense-retrieval baseline over lexicon rows, plus a passage-retrieval variant over native Wikipedia where one exists. If passages beat glosses, the lesson is that descriptive context matters more than lexical mapping, and the "label only" ablation will tell us which part of the gloss does the work.
Why not just fine-tune on a few target-language examples, as xGQA suggests? Forty-eight examples recovering 20 points (xGQA) is strong. However, it needs labeled multimodal data and one training run per language, which does not scale to the thousands of languages the lexicons cover. We will run this few-shot fine-tuning on xGQA languages as a reference. The comparison asks how much of that gain glossing reaches for free.
Decisive Checks
The first check is on xGQA across its 7 languages with LLaVA-1.5 and PALO. The question is whether glossing recovers a meaningful share of the English–local gap, with gains rising in coverage-stratified bins. A flat curve across coverage bins falsifies the lexical account.
The second, the main test, is on CVQA across its 31 languages, stratified by resource level and by whether the item is entity-bound. Glossing should beat translate-test on the low-resource, entity-bound subset. Keyed lookup should beat dense retrieval over identical rows. The random-gloss control should show no gain.
The third check covers IndicVisionBench VQA, including Odia. It tests whether gains hold where scripts are weakly supported, and whether glosses help the model reject false cultural premises in the adversarial items.
The diagnostic check takes items with full gloss coverage that the model still gets wrong in the local language but right in English. We will annotate those residual errors. If most are reasoning errors rather than vocabulary errors, the native-language penalty contains a component that no lookup can fix. The proposal would then have mapped where lexical grounding ends and language-conditioned reasoning begins.
References
- CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
- VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding
- Connecting Language Technologies with Rich, Diverse Data Sources Covering Thousands of Languages
- Converting Legacy Data to CLDF: A FAIR Exit Strategy for Linguistic Web Apps
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
- xGQA: Cross-Lingual Visual Question Answering
- Palo: A Polyglot Large Multimodal Model for 5B People
- Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
- A Common Framework for Syntactic Annotation
- Pivot Through English: Reliably Answering Multilingual Questions without Document Retrieval
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- Tagging Ingush - Language Technology For Low-Resource Languages Using Resources From Linguistic Field Work
- SHR++: An Interface for Morpho-syntactic Annotation of Sanskrit Corpora