Systems that answer questions about images using outside knowledge, such as "when was this bridge built?" or "what does this bird eat?", mostly fail at one step: deciding which encyclopedia entry the picture shows. Once that entity is fixed correctly, the rest is comparatively easy. On INFOSEEK, giving the reader model the correct entity raises accuracy from 18.2% to 45.6% (Hu et al. 2023). On Encyclopedic VQA, PaLM scores 48.8% with automatic retrieval and 87.0% when it is handed the correct knowledge-base section (Mensink et al. 2023). This proposal borrows an old idea from word sense disambiguation, the task of deciding which meaning of an ambiguous word is intended. That idea is that unrelated languages rarely share the same ambiguities. English crane covers a bird and a machine. Chinese uses 鹤 for one and 起重机 for the other.
The proposal is simple. We ask a multimodal model to name the pictured entity in several typologically distant languages. Each name is resolved against that language's labels in a multilingual knowledge base, and an entity is accepted only if the names agree on it. A vernacular name that is ambiguous in English, or an English-biased guess, then has to survive cross-checks against Japanese, Russian, or Arabic naming. In those languages the mapping from name to entity breaks in different places.
Core Claim
The claim is this: agreement across languages is a recognition-verification signal that is cheap and independent of retrieval, and it fixes a measurable fraction of entity errors in knowledge-based visual question answering. The gain should grow with the typological distance between the languages used, because the benefit comes from the languages having uncorrelated ambiguity. Two lines of prior work found this distance effect in text-only and static-ontology settings. One shows that pairing distant languages gives a stricter sense filter (Wang; Aziz et al.). The other is MSGI, which found that English plus Chinese helps sememe prediction more than English plus French (Qi et al. 2022). Nobody has tested whether the effect carries over to live visual entity recognition, where the "translation" is produced by the same model that looked at the picture.
The idea also has a clear way to fail. If the multimodal model names things in Chinese by internally translating its English guess, its errors across languages will be perfectly correlated and the signal adds nothing. Measuring that correlation is the first experiment, and the whole proposal depends on its result.
Research Program
The pipeline starts from an existing entity retriever. EntRAG's EntBind binds entity tokens to visual features and then reranks the candidates (EntRAG), and it returns its top-$K$ candidate entities. Separately, we prompt a multimodal model to give the entity's name in each language $l$ from a fixed set $L$, for example English, Chinese, Japanese, Russian, Arabic, and Hindi. Each generated name is linked to Wikidata identifiers. Two linkers will be compared. The first is exact or alias matching against Wikidata labels in language $l$. The second is constrained decoding into a prefix tree of valid entity names, as mGENRE does over 89 million multilingual names (De Cao et al. 2021).
mGENRE is text-only and never sees an image. We reuse its constrained-decoding machinery, and we reuse its treatment of the name's language as a variable to sum over. Its contribution here is to turn a generated string into a distribution over entities, not to look at pixels.
Candidates are rescored using the soft combination that Luan et al. 2020 introduced for text WSD (their SOFTCONSTRAINT product of experts). We adopt that over their HARDCONSTRAINT filter because hard filters drop correct answers whenever a translation is wrong. The combined score is
$$s(e) = \log p_{\text{ret}}(e \mid x) + \sum_{l \in L} \lambda_l \log\big(\epsilon + p_l(e \mid n_l)\big),$$
where $e$ is a candidate entity, $x$ the image and question, $p_{\text{ret}}$ the base retriever's normalized score, $n_l$ the name generated in language $l$, $p_l(e \mid n_l)$ the probability the linker assigns to $e$ given that name, $\lambda_l$ a per-language weight tuned on development data, and $\epsilon$ a floor that keeps a missing label from vetoing a candidate. The floor matters because many long-tail entities have no label in some languages. Navigli & Ponzetto 2010 report that machine-translated BabelNet lexicalizations are only about 72% precise, which is why we take labels from Wikidata rather than from translations. The top-scoring entity's knowledge section is then passed to the same answer generator EntRAG uses, so any change in the final answer comes from the entity choice.
Several neighbouring systems do related things without testing this. VP-MEL's Detective-VLM generates an English description of a boxed object to help link it (Mi et al. 2025). We generate names across languages and use their disagreement as evidence, which VP-MEL never does. Cross-lingual visual verb sense disambiguation uses images to choose among translations (NAACL 2019), the reverse direction from ours. MSGI combines multilingual glosses and images to complete a static ontology offline, not to verify a recognition at question time. Language pivoting for Latin (Ghinassi et al. 2024) shows that intersecting senses across a pivot language beats trusting one projection. That result constrains our design toward agreement-based scoring.
Visual Riddles found a 32% reasoning gap that remains after the clue is recognized, so better recognition will not fix everything. We therefore report entity accuracy separately from answer accuracy.
Baselines
The naive reference point is the base retriever's top-1 entity with no rescoring. We use EntRAG's retrieval and reranking stages unchanged, reproduced on the same knowledge base and with the same answer generator. EntRAG is also the strongest existing method in the file: it reports 46.1 accuracy on E-VQA and 44.5 on InfoSeek (EntRAG). Our numbers will come from our own reproduction, and we will report the gap to these published figures. Beating it means a statistically significant gain in end-to-end answer accuracy, with the same reader, on both benchmarks. On E-VQA the reference ceiling for imperfect automatic retrieval is PaLM with Google Lens at 48.8% (Mensink et al. 2023). That system uses a proprietary retriever, so it is a context number and not a like-for-like competitor.
The key ablation is English-only self-consistency. The multimodal model generates the same number of names, $|L|$ samples at the same temperature, all in English, and they are scored with the same equation. This matches the generation compute and the tuning budget for $\lambda$ exactly, and it isolates the claim that language diversity, not just extra samples, carries the signal. A second ablation varies typological distance at a fixed number of languages, comparing English with French, Spanish and German against English with Chinese, Japanese and Russian. This tests the predicted distance effect. A third ablation swaps the HARDCONSTRAINT-style intersection in for the soft score to measure how often hard filtering kills correct entities. The oracle-entity setting is the upper bound: 45.6% for FiD on INFOSEEK-Human (Hu et al. 2023).
Why not just…?
Why not just sample more English answers and vote? This is the most serious threat, because our method also takes several samples. If the model's errors in different languages are as correlated as its errors across English samples, multilingual voting reduces to ordinary self-consistency. The compute-matched ablation above settles this directly. We will also report the pairwise error correlation between languages. If English self-consistency matches the multilingual score, we will report the idea as negative.
Why not just ask for the scientific name or the Wikidata identifier? For species, a Latin binomial is close to an unambiguous pivot. This alternative could plausibly match our method on the iNaturalist half of E-VQA, so we include it as a baseline and break results out by domain. Landmarks, artifacts and people have no such canonical name. There, multilingual labels are the only independent naming systems available, and we expect the gap to show up on the landmark half. Asking for an identifier directly invites made-up identifiers. We include it anyway as a further baseline.
Why not just improve the visual retriever? Better image-entity binding is complementary, not competing. EntBind improved Top-1 Recall by only 1.3% on E-VQA, which suggests visual similarity is saturating among near-identical sibling entities. Our rescoring sits on top of any retriever. We will test it on two retrievers, EntRAG and a CLIP baseline, to show the gain does not depend on a weak base.
Why not just use a larger closed-book model? PaLI-X scores 10.8% on INFOSEEK-Human (Hu et al. 2023), and PaLI falls from 64.5% on OK-VQA to 13.0% on E-VQA (Mensink et al. 2023). Scale alone does not solve long-tail recognition. Our method uses the large model as a namer, not as the source of the answer.
Proposed Experiments
The first, decisive check runs before any end-to-end evaluation. On a few thousand E-VQA and InfoSeek development images, we generate names in every language, link each name, and compute the correlation of entity errors across language pairs. The proposal goes forward only if the errors of distant language pairs are measurably less correlated than those of English resamples.
Main results follow on E-VQA and InfoSeek, with answer accuracy and entity top-1 accuracy, split into species and landmark questions. We also build a diagnostic subset from Wikidata of entities whose English label or alias is shared by another entity among the top-$K$ candidates. This is where the mechanism should help most, and we predict at least double the average gain there.
For transfer to entity linking without a question, we evaluate on VPWiki. The strongest reported number there is IIER at 48.36% Hit@1 (Mi et al. 2025), and we will add our rescoring on top of an IIER-style candidate list. We found no reported result on Lacuna for multilingual-naming verification on any of these benchmarks.
Finally, we plot the gain against the number of languages and against average typological distance. We fit the distance trend only if the first check shows decorrelated errors. The prediction that distance matters comes from the text-only results cited above, and a flat curve would show that their effect does not transfer to model-generated names.
References
- Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
- Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
- Automatic Acquisition of English Topic Signatures Based on a Second Language
- Multilingual WSD-like Constraints for Paraphrase Extraction
- Sememe Prediction for BabelNet Synsets using Multilingual and Multimodal Information
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Multilingual Autoregressive Entity Linking
- Improving Word Sense Disambiguation with Translations
- BabelNet: Building a Very Large Multilingual Semantic Network
- VP-MEL: Visual Prompts Guided Multimodal Entity Linking
- Cross-lingual Visual Verb Sense Disambiguation
- Language Pivoting from Parallel Corpora for Word Sense Disambiguation of Historical Languages: A Case Study on Latin
- Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models