TL;DR
VEPO (Visual Evidence-guided Preference Optimization) is a novel training-time method designed to reduce object hallucinations in Large Vision-Language Models (LVLMs), particularly for low-resource languages. By using English-guided attention maps to identify and mask key visual evidence, VEPO creates preference pairs that teach models to rely on actual image content rather than linguistic shortcuts, leading to massive performance gains in languages like Hindi without requiring external translation models.
What Problem Does This Paper Address?
While Large Vision-Language Models (LVLMs) like LLaVA or Qwen-VL are impressive, they frequently "hallucinate"—claiming an object exists in an image when it doesn't. This problem is significantly worse in low-resource languages (e.g., Hindi or Arabic) compared to English.
The authors identify this as multilingual hallucination. Because these models have weaker cross-modal alignment for non-English languages, they tend to ignore the image and rely on "language priors"—essentially guessing based on common word associations. As shown in the paper's opening analysis, there is a stark performance gap between high-resource and low-resource queries on standard benchmarks.
Existing solutions often rely on expensive human annotations, external translation models, or inference-time interventions that can cause "language drift" (e.g., a model answering a Hindi question in English because it's trying to mimic English reasoning).
Methods
VEPO introduces a three-stage training pipeline that functions entirely within the LVLM's existing architecture:
- Key Visual Evidence Localization: The researchers use English as a "high-resource probe." Since English grounding is typically strong, they extract mid-layer cross-modal attention weights (specifically layers 11–14 in LLaVA-1.5) to see which image patches the model focuses on to answer a question correctly in English.
- Key Evidence Occlusion: They create a "rejected" version of the image by masking the top 20% of the most-attended patches. This creates a version of the image where the critical evidence for the answer is missing, but the overall scene context remains.
- Evidence-Sufficiency Preference Optimization: Using Direct Preference Optimization (DPO), the model is trained on low-resource language queries. Crucially, instead of comparing a "correct" answer to an "incorrect" one, VEPO compares the same correct answer under two different image conditions: the original image (chosen) vs. the masked image (rejected).
This approach forces the model to realize that the answer is only valid when the specific visual evidence is present, effectively "hardening" its visual grounding.
Datasets
- Training: The authors utilized the PALO multilingual visual-instruction corpus, which contains approximately 2.1 million instructions across 10 languages.
- Evaluation:
- POPE: A standard benchmark for object hallucination across various splits (Random, Popular, Adversarial).
- MME: A comprehensive benchmark for perception and cognition.
- Languages Studied: Chinese (Zh), French (Fr), and Hindi (Hi) were used as representative mid- to low-resource settings.
Key Insights and Ideas
- The "English Probe" Advantage: The paper cleverly exploits the fact that LVLMs already "know" how to look at images when prompted in English. VEPO transfers this internal knowledge to other languages during training without needing a separate teacher model.
- Evidence vs. Answer Preference: Standard DPO (comparing text answers) can be "cheated" by a model that just learns better language patterns. By keeping the text answer constant and varying the image evidence, VEPO directly targets the vision-language bottleneck.
- Layer Selection Matters: The authors found a "sweet spot" in mid-layers (11-14). Shallow layers focus too much on local textures, while deep layers are too biased toward final text generation. Mid-layers provide the most stable semantic localization.
- Cross-Lingual Regularization: For extremely low-resource languages like Hindi, training solely on Hindi pairs can be unstable. Mixing in English preference pairs at a 1:1 ratio acts as a stabilizer, preventing the model's vision-language alignment from collapsing.
Key Results
VEPO achieved significant improvements across all tested languages and benchmarks:
- POPE Hallucination Reduction: On the pretrained backbone, Hindi POPE average F1 scores skyrocketed from 0.302 to 0.644. Chinese and French also saw healthy gains (+3.5 and +5.2 F1 points, respectively).
- Perception Gains: In the MME benchmark, total scores rose for all languages, with the most notable improvements in perception tasks like object existence, color, and position.
- Evidence-Guided vs. Random: Ablation studies confirmed that masking specific evidence identified by attention is far more effective than random masking, proving that the model is learning to link specific regions to specific concepts.
Limitations and Future Work
- English Dependency: The method relies on the model having at least one high-resource language (English) with strong grounding to act as a probe. If a model is poorly grounded even in English, VEPO's localization might fail.
- Architecture Specificity: The specific mid-layers used for attention extraction (11-14) were validated for LLaVA-1.5. Other architectures (like those using different vision encoders or deeper LLMs) might require retuning these indices.
- Future Directions: The authors suggest exploring ways to identify key evidence without relying on English probes and extending the method to even more diverse, extremely low-resource languages.
Summary updated