Visual Hallucinations of Multi-modal Large Language Models

Wen Huang, Hongbin Liu, Minxin Guo, Neil Zhenqiang Gong

Abstract

Visual hallucination (VH) occurs when a multi-modal Large Language Model (MLLM) imagines incorrect details about an image during visual question answering. Existing studies often rely on standard image datasets, which leads to a biased understanding of MLLM performance due to limited diversity and potential data contamination. In this work, we propose VHTest, a tool designed to generate a diverse set of VH instances. VHTest identifies initial VH instances in existing datasets, generates text descriptions for specific VH modes, and utilizes text-to-image generative models like DALL-E 3 to create targeted "hallucination traps." We introduce a benchmark dataset of 1,200 VH instances across eight modes. Our evaluation reveals that state-of-the-art models, including GPT-4V, LLaVA-1.5, and MiniGPT-v2, hallucinate on a significant portion of these instances. Furthermore, we demonstrate that fine-tuning MLLMs on our benchmark reduces hallucination rates without compromising general performance.

Introduction: The Challenge of Visual Hallucination

Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities in integrating vision and language. However, they frequently suffer from visual hallucinations (VH)—instances where the model describes objects or attributes that are not present in the visual input. For example, a model might miscount objects in a room or misidentify the spatial relationship between items.

An example of a visual hallucination where GPT-4V incorrectly counts the number of lamps in a scene.

Addressing VH is difficult because current benchmarks often rely on standard datasets like COCO. These datasets suffer from two main issues: limited diversity, failing to cover the full spectrum of possible visual errors, and data contamination, as these common images were likely included in the models' original training sets. This makes it unclear whether a model is truly "seeing" the image or simply recalling memorized patterns.

Methodology: The VHTest Pipeline

To overcome these limitations, we developed VHTest, an adversarial pipeline that systematically identifies and exploits model weaknesses. The process consists of three main stages:

  1. Identifying Blind Spots: We look for discrepancies between different vision encoders. Specifically, we identify image pairs that CLIP (a common encoder for semantic alignment) perceives as nearly identical, but DINO v2 (which captures finer spatial details) recognizes as distinct. These contradictions highlight semantic blind spots where an MLLM is likely to fail.
  2. Failure Analysis: We use an MLLM to analyze these failure cases and describe the specific visual features—such as overlapping objects or complex textures—that trigger the confusion.
  3. Adversarial Generation: These descriptions are fed into DALL-E 3 to generate brand-new, synthetic images designed to be "hallucination traps." This ensures the benchmark is both diverse and free from the contamination found in traditional datasets.

The VHTest Benchmark

Using this pipeline, we curated a benchmark of 1,200 VH instances categorized into eight distinct "modes" to test different aspects of visual reasoning:

  • Individual Properties: Existence, Shape, Color, Orientation, and OCR (Optical Character Recognition).
  • Group Properties: Size, Position, and Counting.

These categories allow for a granular analysis of where specific models struggle most, moving beyond simple object detection to complex relational and attribute-based reasoning.

Benchmark and Results

Our evaluation of leading MLLMs reveals a significant reliability gap. Even the most advanced models struggle with the adversarial nature of the VHTest dataset.

Comparison of model responses across different hallucination modes in the VHTest benchmark.

Key findings from our testing include:

  • Low Overall Accuracy: On our Open-Ended Question (OEQ) benchmark, GPT-4V achieved only 38.3% accuracy. LLaVA-1.5 and MiniGPT-v2 performed significantly worse, scoring 22.9% and 7.5%, respectively.
  • Challenging Modes: Orientation, Counting, and OCR proved to be the most difficult categories across all models. For orientation tasks, the average accuracy dropped to a mere 14%.
  • Effective Mitigation: We found that the VHTest benchmark is not just an evaluation tool but also a powerful training resource. Fine-tuning LLaVA-1.5 on a subset of this adversarial data improved its accuracy in the "Position" category from 33.3% to 53.3% without negatively impacting its performance on standard benchmarks.

Conclusion and Future Work

VHTest provides a rigorous framework for identifying and mitigating visual hallucinations in MLLMs. By using generative AI to find the weaknesses of other AI systems, we can create more robust and reliable models. While the current pipeline requires some human verification for final question-answer pairs, future work will focus on full automation to scale the production of hallucination-proof training data. As MLLMs move into high-stakes applications like autonomous driving and medical diagnostics, benchmarks that prioritize adversarial robustness will be essential for ensuring safety and reliability.

Summary updated

Abstract

Visual hallucination (VH) means that a multi-modal LLM (MLLM) imagines incorrect details about an image in visual question answering. Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs' performance under VH due to limited diversity of such VH instances. In this work, we propose a tool called VHTest to generate a diverse set of VH instances. Specifically, VHTest finds some initial VH instances in existing image datasets (e.g., COCO), generates a text description for each VH mode, and uses a text-to-image generative model (e.g., DALL-E-3) to generate VH images based on the text descriptions. We collect a benchmark dataset with 1,200 VH instances in 8 VH modes using VHTest. We find that existing MLLMs such as GPT-4V, LLaVA-1.5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark. Moreover, we find that fine-tuning an MLLM using our benchmark dataset reduces its likelihood to hallucinate without sacrificing its performance on other benchmarks. Our benchmarks are publicly available: https://github.com/wenhuang2000/VHTest.
BibTeX
@inproceedings{huang2024visual,
  author = {Wen Huang and Hongbin Liu and Minxin Guo and Neil Zhenqiang Gong},
  title = {{Visual Hallucinations of Multi-modal Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=geT8UzNO6Q},
}
Figure