VEPO: Mitigating Multilingual Hallucinations in LVLMs via Visual Evidence-guided Preference Optimization

Chengpeng Fu, Xiaocheng Feng, Wenshuai Huo, Yichong Huang, Baohang Li, Duyu Tang, Ting LiuarXiv 2026 · arXiv

TL;DR

VEPO (Visual Evidence-guided Preference Optimization) is a novel training-time method designed to reduce object hallucinations in Large Vision-Language Models (LVLMs), particularly for low-resource languages. By using English-guided attention maps to identify and mask key visual evidence, VEPO creates preference pairs that teach models to rely on actual image content rather than linguistic shortcuts, leading to massive performance gains in languages like Hindi without requiring external translation models.

What Problem Does This Paper Address?

While Large Vision-Language Models (LVLMs) like LLaVA or Qwen-VL are impressive, they frequently "hallucinate"—claiming an object exists in an image when it doesn't. This problem is significantly worse in low-resource languages (e.g., Hindi or Arabic) compared to English.

The authors identify this as multilingual hallucination. Because these models have weaker cross-modal alignment for non-English languages, they tend to ignore the image and rely on "language priors"—essentially guessing based on common word associations. As shown in the paper's opening analysis, there is a stark performance gap between high-resource and low-resource queries on standard benchmarks.

Figure 1: Multilingual performance gap of Qwen-VL-Chat on POPE-COCO: non-English, especially low-resource, lan- guages lag behind English.

Existing solutions often rely on expensive human annotations, external translation models, or inference-time interventions that can cause "language drift" (e.g., a model answering a Hindi question in English because it's trying to mimic English reasoning).

Methods

VEPO introduces a three-stage training pipeline that functions entirely within the LVLM's existing architecture:

  1. Key Visual Evidence Localization: The researchers use English as a "high-resource probe." Since English grounding is typically strong, they extract mid-layer cross-modal attention weights (specifically layers 11–14 in LLaVA-1.5) to see which image patches the model focuses on to answer a question correctly in English.
  2. Key Evidence Occlusion: They create a "rejected" version of the image by masking the top 20% of the most-attended patches. This creates a version of the image where the critical evidence for the answer is missing, but the overall scene context remains.
  3. Evidence-Sufficiency Preference Optimization: Using Direct Preference Optimization (DPO), the model is trained on low-resource language queries. Crucially, instead of comparing a "correct" answer to an "incorrect" one, VEPO compares the same correct answer under two different image conditions: the original image (chosen) vs. the masked image (rejected).
Figure 2: Constructing a key-evidence masked image from English attention. Mid-layer attention from the last query token ranks image patches; the top-ρ patches are blacked out to form the rejected image, while the original image remains the chosen image.

This approach forces the model to realize that the answer is only valid when the specific visual evidence is present, effectively "hardening" its visual grounding.

Datasets

  • Training: The authors utilized the PALO multilingual visual-instruction corpus, which contains approximately 2.1 million instructions across 10 languages.
  • Evaluation:
    • POPE: A standard benchmark for object hallucination across various splits (Random, Popular, Adversarial).
  • MME: A comprehensive benchmark for perception and cognition.
  • Languages Studied: Chinese (Zh), French (Fr), and Hindi (Hi) were used as representative mid- to low-resource settings.

Key Insights and Ideas

  • The "English Probe" Advantage: The paper cleverly exploits the fact that LVLMs already "know" how to look at images when prompted in English. VEPO transfers this internal knowledge to other languages during training without needing a separate teacher model.
  • Evidence vs. Answer Preference: Standard DPO (comparing text answers) can be "cheated" by a model that just learns better language patterns. By keeping the text answer constant and varying the image evidence, VEPO directly targets the vision-language bottleneck.
  • Layer Selection Matters: The authors found a "sweet spot" in mid-layers (11-14). Shallow layers focus too much on local textures, while deep layers are too biased toward final text generation. Mid-layers provide the most stable semantic localization.
  • Cross-Lingual Regularization: For extremely low-resource languages like Hindi, training solely on Hindi pairs can be unstable. Mixing in English preference pairs at a 1:1 ratio acts as a stabilizer, preventing the model's vision-language alignment from collapsing.

Key Results

VEPO achieved significant improvements across all tested languages and benchmarks:

  • POPE Hallucination Reduction: On the pretrained backbone, Hindi POPE average F1 scores skyrocketed from 0.302 to 0.644. Chinese and French also saw healthy gains (+3.5 and +5.2 F1 points, respectively).
  • Perception Gains: In the MME benchmark, total scores rose for all languages, with the most notable improvements in perception tasks like object existence, color, and position.
  • Evidence-Guided vs. Random: Ablation studies confirmed that masking specific evidence identified by attention is far more effective than random masking, proving that the model is learning to link specific regions to specific concepts.
Figure 3: Comparison between VEPO (left) and standard multimodal DPO (right). VEPO contrasts evidence-sufficient vs. evidence-missing images under the same correct answer; standard DPO contrasts different answers under the same image. The training examples illustrated in the figure are presented in Chinese, and the semantic content of the questions is consistent with that shown in Figure 2.

Limitations and Future Work

  • English Dependency: The method relies on the model having at least one high-resource language (English) with strong grounding to act as a probe. If a model is poorly grounded even in English, VEPO's localization might fail.
  • Architecture Specificity: The specific mid-layers used for attention extraction (11-14) were validated for LLaVA-1.5. Other architectures (like those using different vision encoders or deeper LLMs) might require retuning these indices.
  • Future Directions: The authors suggest exploring ways to identify key evidence without relying on English probes and extending the method to even more diverse, extremely low-resource languages.

Summary updated

Abstract

Large vision-language models (LVLMs) achieve strong multimodal performance, yet they remain prone to object hallucination---generating answers that are linguistically fluent but visually ungrounded. This issue is especially severe in low-resource languages, where weaker cross-modal alignment causes models to rely more on language priors than on image evidence. Existing remedies either require costly multilingual annotations or external translation/reward models, or intervene at inference time in ways that can distort the output language. We propose VEPO(Visual Evidence-guided Preference Optimization), a training-time method that transfers reliable visual grounding from English to other languages without external models. VEPO first localizes key visual evidence using English queries via mid-layer cross-modal attention, then constructs preference pairs that share the same correct answer but differ in evidence sufficiency (full image vs.\ key-region masked image). Direct preference optimization on these pairs encourages low-resource inputs to prefer answers supported by complete visual evidence. Experiments on multilingual POPE and MME with Chinese, French, and Hindi show consistent hallucination reduction and perception gains, with the largest improvements on the lowest-resource language. Analyses further confirm that VEPO sharpens question-relevant attention and that evidence-guided masking outperforms random masking.
BibTeX
@misc{fu2026vepo,
  author = {Chengpeng Fu and Xiaocheng Feng and Wenshuai Huo and Yichong Huang and Baohang Li and Duyu Tang and Ting Liu},
  title = {{VEPO: Mitigating Multilingual Hallucinations in LVLMs via Visual Evidence-guided Preference Optimization}},
  year = {2026},
  eprint = {2611.18096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18096},
}
Figure