Abstract
Large Vision-Language Models (LVLMs) have demonstrated remarkable performance in performing complex multimodal tasks. However, they are still plagued by object hallucination: the misidentification or misclassification of objects present in images. To this end, we propose HALLUCINOGEN, a novel visual question answering (VQA) object hallucination attack benchmark that utilizes diverse contextual reasoning prompts to evaluate object hallucination in state-of-the-art LVLMs. We design a series of contextual reasoning hallucination prompts to evaluate LVLMs' ability to accurately identify objects in a target image while asking them to perform diverse visual-language tasks such as identifying, locating or performing visual reasoning around specific objects. Further, we extend our benchmark to high-stakes medical applications and introduce MED-HALLUCINOGEN, hallucination attacks tailored to the biomedical domain, and evaluate the hallucination performance of LVLMs on medical images, a critical area where precision is crucial. Finally, we conduct extensive evaluations of eight LVLMs and two hallucination mitigation strategies across multiple datasets to show that current generic and medical LVLMs remain susceptible to hallucination attacks.
Introduction: The Challenge of Implicit Hallucination
Large Vision-Language Models (LVLMs) have reached a level of capability where they can often correctly identify objects when asked directly. However, a significant vulnerability remains: object hallucination. Most existing benchmarks evaluate this via explicit, closed-form attacks—for instance, asking, "Is there a car in this image?" While modern models excel at these simple detection tasks, they frequently fail when faced with implicit reasoning.
The core issue is the "Implicit Trap." If a prompt assumes an object exists (e.g., "Describe the area around the car" when no car is present), models often prioritize linguistic helpfulness over visual accuracy. Instead of correcting the user, they fabricate detailed descriptions of non-existent surroundings, essentially hallucinating to satisfy the prompt's premise.
The HALLUCINOGEN Benchmark
To address these limitations, we introduced HALLUCINOGEN, a framework that shifts the focus from direct questioning to implicit hallucination attacks. The benchmark categorizes visual entities into two distinct types:
- Salient Entities: Prominent, everyday objects like cars or animals, sourced from the MS-COCO dataset.
- Latent Entities: Features requiring specialized domain knowledge, such as medical conditions in chest X-rays (forming the MED-HALLUCINOGEN subset).
The benchmark utilizes 90,000 image-prompt pairs to test models across three levels of increasing difficulty:
- Localization (LOC): Pinpointing the position of a non-existent object.
- Visual Context (VC): Describing the environment surrounding a non-existent object.
- Counterfactual Reasoning (CF): Imagining scene changes if a non-existent object were removed.
To evaluate responses, we employed a combination of automated string-matching and an "LLM-as-a-judge" approach using GPT-4o to detect if a model's response incorrectly assumed the presence of the target entity.
Key Insights: Why Models Hallucinate
Our analysis revealed several critical reasons for model failure:
- Affirmative Bias: Models exhibit a strong tendency to agree with the prompt's assumptions. As task difficulty increases, the rate of affirmative errors (saying "Yes" to a hallucination) can rise as high as 96.2%.
- Chain-of-Thought (CoT) Backfires: While CoT usually aids reasoning, in hallucination tasks, it provides models with more "space" to construct elaborate, self-consistent lies, actually increasing hallucination rates.
- Language Over Vision: Attention map analysis shows that during hallucinations, models focus heavily on text tokens in the prompt and their own generated text, largely ignoring the actual visual evidence from the image.
Experimental Results
We evaluated eight state-of-the-art LVLMs, including Gemini and Llama-3.2-VL. The results indicate a pervasive susceptibility to implicit attacks.
Key findings include:
- Performance Degradation: Accuracy dropped by 6.8% to 29.0% when moving from explicit to implicit attacks.
- Medical Domain Vulnerability: On latent medical entities, most models performed near the level of random guessing (~50% accuracy). This suggests a lack of "visual common sense" necessary to admit the absence of a specific condition.
- Mitigation Ineffectiveness: Current strategies like RLHF and LURE failed to provide a robust defense. For example, RLHF-tuned models saw accuracy drops of over 21% when progressing to complex counterfactual tasks.
Conclusion and Future Work
The HALLUCINOGEN benchmark demonstrates that current LVLMs are highly vulnerable to sophisticated linguistic prompts that assume the presence of non-existent objects. This is especially concerning in high-stakes domains like healthcare. Future research must focus on developing training techniques that force models to prioritize visual grounding over linguistic priors and expanding the benchmark to include more complex modalities like video understanding.
Summary updated