Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructions

Zhi-Jie Tan, Yuzhi Li, Shengwei Meng, Xiang Yuan, Weiping Li, Tong Mo, Bingce Wang, Xu ChuarXiv 2025 · arXiv

TL;DR

This paper tackles the problem of "hallucinations on object attributes" (HOOA) in Large Vision-Language Models (LVLMs), where models incorrectly describe visual details. They propose MIAVLM, a novel model that uses multiple viewpoints of an object as input and a special "Multiview Attributes Perceiver" to handle these views without being sensitive to their order. Additionally, they introduce negative instructions to curb the models' tendency to always say "Yes."

What Problem Does This Paper Address?

Large Vision-Language Models (LVLMs) are powerful tools that can understand and generate text based on images. However, they often "hallucinate," meaning they produce outputs that are inconsistent with the input image. This paper focuses on a specific type of hallucination: Hallucination on Object Attributes (HOOA). This occurs when an LVLM incorrectly describes an object's characteristics, like saying a red apple is green. Existing benchmarks often bundle HOOA with other hallucination types (object existence and relationships), making it hard to isolate and address HOOA specifically. The authors argue that this is a critical issue, especially as LVLMs are increasingly used in applications requiring accurate visual understanding.

Methods

The core of the proposed solution is the Multiview Image Augmented VLM (MIAVLM). Here's a breakdown of its key components:

  • Multiview Image Generation: Instead of relying on a single image, MIAVLM leverages advancements in 3D generation to create multiple viewpoints of an object from a single input image. These multiview images act as richer visual prompts.
  • Multiview Attributes Perceiver (MAP): This novel submodule is designed to process these multiple image inputs. It addresses two key challenges:
  • Input Order Insensitivity: LVLMs, like LLMs, can be sensitive to the order of input prompts. MAP is designed to process multiview images in a way that their order doesn't affect the final output, making the model more robust.
  • Information Alignment: MAP effectively aligns the visual information from the different viewpoints with the Large Language Model (LLM) component.
  • Negative Instructions: The authors observed that LVLMs have a bias towards answering "Yes." To counteract this, they designed "negative questions" (questions where the correct answer is "No") and used them as negative instructions during training. This helps the model learn to provide more balanced and accurate "Yes/No" responses.

Datasets

  • CelebAText-HQ [16]: This dataset, manually annotated with facial attributes, was used to construct the authors' new benchmark.
  • HoOA Benchmark (New): The authors created a new benchmark specifically for evaluating HOOA. They transformed descriptions from CelebAText-HQ into questions, generating both "positive" (answer "Yes") and "negative" (answer "No") question sets. This benchmark aims to isolate HOOA from other hallucination types.

Key Insights and Ideas

  • Multiview is Key for Fine-Grained Attributes: Providing LVLMs with multiple perspectives of an object significantly enhances their ability to accurately describe fine-grained attributes, overcoming limitations of single-image views.
  • Order Matters (for other models): The order in which multiview images are presented can significantly impact the performance of existing LVLMs, highlighting the need for order-invariant processing.
  • Negative Instructions are Crucial: LVLMs' inherent bias towards "Yes" answers can be effectively mitigated by training them with negative instructions, leading to more balanced and accurate responses.
  • 3D Generation as a Data Augmentation Tool: Leveraging 3D generation techniques to create synthetic multiview data is a promising avenue for improving LVLM robustness and reducing hallucinations.

Key Results

  • Improved HOOA Metric: MIAVLM achieved a HOOA metric of 0.775/0.787 (positive/negative accuracy) using single images and 0.762/0.812/0.787 (positive/negative/overall HOOA metric) with 9 multiview images, outperforming other compared LVLMs.
  • Efficiency: MIAVLM is noted for its efficiency, with a significantly lower parameter count (1.0B) and inference time (0.071s/0.105s) compared to many other models.
  • Order Insensitivity: MIAVLM demonstrated robustness to the order of multiview inputs, unlike OpenFlamingo models, which showed significant performance variations based on input order.
  • Negative Instructions Boost Performance: Using negative instructions improved negative accuracy by a substantial margin (from 0.540 to 0.812 for MIAVLM without NI vs. with NI), though it slightly decreased positive accuracy.

Limitations and Future Work

  • 3D Generation Dependency: The method relies on the quality of 3D generation. If the generated 3D models or their renderings are poor, it could negatively impact performance.
  • Computational Cost of 3D Generation: While MIAVLM itself is efficient, the initial step of generating 3D representations and multiview images can be computationally intensive.
  • Future Work: The authors suggest that their approach of using multiview images and negative instructions can be extended to other types of hallucinations (HoOE and HoOR) and applied to a wider range of LVLM architectures. Further research into more efficient 3D generation techniques and more sophisticated ways to incorporate negative feedback could also be beneficial.

Summary updated

Abstract

Current popular Large Vision-Language Models (LVLMs) are suffering from Hallucinations on Object Attributes (HoOA), leading to incorrect determination of fine-grained attributes in the input images. Leveraging significant advancements in 3D generation from a single image, this paper proposes a novel method to mitigate HoOA in LVLMs. This method utilizes multiview images sampled from generated 3D representations as visual prompts for LVLMs, thereby providing more visual information from other viewpoints. Furthermore, we observe the input order of multiple multiview images significantly affects the performance of LVLMs. Consequently, we have devised Multiview Image Augmented VLM (MIAVLM), incorporating a Multiview Attributes Perceiver (MAP) submodule capable of simultaneously eliminating the influence of input image order and aligning visual information from multiview images with Large Language Models (LLMs). Besides, we designed and employed negative instructions to mitigate LVLMs’ bias towards "Yes" responses. Comprehensive experiments demonstrate the effectiveness of our method.
BibTeX
@misc{tan2025mitigating,
  author = {Zhi-Jie Tan and Yuzhi Li and Shengwei Meng and Xiang Yuan and Weiping Li and Tong Mo and Bingce Wang and Xu Chu},
  title = {{Mitigating Hallucinations on Object Attributes using Multiview Images and Negative Instructions}},
  year = {2025},
  eprint = {2501.10011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.10011},
  howpublished = {arXiv},
}
Figure