HallE-Control: Controlling Object Hallucination in Large Multimodal Models

Bohan Zhai, Shijia Yang, Xiangchen Zhao, Chenfeng Xu, Sheng Shen, Dongdi Zhao, Kurt Keutzer

Abstract

Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, particularly in detailed captioning. To address this, we introduce CCEval, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we investigate such hallucination from different aspects, including image resolution, language decoder size, and instruction data quality. Our findings underscore the unwarranted inference when language descriptions include details at a finer granularity than what the vision module can ground, thus inducing hallucination. To control this, we introduce HallE-Control, a controllable LMM that can shift between exclusively depicting contextual knowledge for grounded objects and blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA-7B while maintaining object coverage.

Introduction: The "Honesty" Problem in LMMs

Large Multimodal Models (LMMs) like LLaVA are increasingly capable of generating rich, descriptive captions for complex scenes. However, they suffer from a persistent "honesty" problem: they frequently hallucinate objects that are not present in the image. This often stems from "unwarranted inference," where the model's internal language weights (parametric knowledge) suggest that certain objects should be present based on context, even if they are occluded or absent. For instance, seeing a dining table might lead a model to describe chairs that are actually out of frame.

The HallE-Control interface allows users to toggle between strict visual grounding and creative inference using a continuous controller.

Existing benchmarks like POPE often fail to capture this because they rely on simple Yes/No questions. When tasked with detailed captioning, models revert to guessing. HallE-Control addresses this by providing a "knob" to regulate this behavior, allowing users to choose between strict adherence to visual evidence and imaginative reasoning.

Methodology

CCEval: A New Standard for Captioning Accuracy

To accurately measure hallucinations in long-form text, the authors developed CCEval (Caption Concept Matching Evaluation). This GPT-4 assisted pipeline extracts objects from a model's detailed caption and compares them against ground-truth annotations. Unlike traditional metrics like CHAIR, CCEval leverages GPT-4’s reasoning to handle synonyms and contextual nuances, providing a more reliable score for both object coverage and hallucination rates.

The HallE-Control Architecture

HallE-Control is built upon the "LM-Switch" technique. The researchers inserted a lightweight, trainable linear layer ($W$) into the word embedding space of a frozen LMM. The training process involves two distinct types of knowledge:

  1. Contextual Knowledge: Objects that the vision encoder can explicitly ground in the image.
  2. Parametric Knowledge: Objects the model "imagines" based on learned word-word associations.
The two-step pipeline for HallE-Control: (1) generating contrastive data with grounded vs. omitted objects, and (2) training a switch layer to shift embeddings based on the control parameter ε.

By training on a contrastive dataset, the model learns to respond to a control parameter $\epsilon$. Setting $\epsilon = -1$ forces the model to stick strictly to what it sees (contextual only), while setting $\epsilon = 1$ allows it to blend in its parametric "imagination."

Key Insights and Results

Resolution vs. Model Size

A striking finding of the study is that increasing image resolution (e.g., using 336px crops or sliding windows) is more effective at reducing hallucinations than scaling up the language decoder. If the vision module (the "eyes") cannot resolve a detail, the language model (the "brain") will inevitably start guessing, regardless of its parameter count.

Data Quality Over Quantity

The research suggests that simply increasing the volume of instruction data can be counterproductive. If the training data contains details that the vision module cannot verify, it encourages the model to hallucinate. High-quality, grounded data is essential for maintaining model honesty.

Performance Metrics

In empirical tests, HallE-Control demonstrated a 44% reduction in object existence hallucinations compared to the LLaVA-7B baseline. Furthermore, it achieved a superior Pareto balance between CHAIR (hallucination rate) and Coverage (the breadth of real objects described). When the "imagination" mode is active, the model can even be trained to wrap its inferences in special tokens (e.g., [object]), providing a built-in confidence signal for the user.

Conclusion

HallE-Control represents a significant step toward more reliable and controllable LMMs. By acknowledging that hallucination is not always "bad"—for example, in robotics where inferring occluded objects is useful—the authors provide a framework to manage it rather than just eliminate it. While currently focused on object existence, this "control knob" approach offers a promising path for addressing attribute and relationship hallucinations in future multimodal research.

Summary updated

Abstract

Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce $\textit{CCEval}$, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from different aspects, including image resolution, the language decoder size, and instruction data amount, quality, granularity. Our findings underscore the unwarranted inference when the language description includes details at a finer object granularity than what the vision module can ground or verify, thus inducing hallucination. To control such hallucinations, we further attribute the reliability of captioning to contextual knowledge (involving only contextually grounded objects) and parametric knowledge (containing inferred objects by the model). Thus, we introduce $\textit{HallE-Control}$, a controllable LMM in terms of $\textbf{Hall}$ucination in object $\textbf{E}$xistence. HallE-Control can condition the captioning to shift between (i) exclusively depicting contextual knowledge for grounded objects and (ii) blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA$_{7B}$ and maintains the object coverage.
BibTeX
@misc{zhai2023halle,
  author = {Bohan Zhai and Shijia Yang and Xiangchen Zhao and Chenfeng Xu and Sheng Shen and Dongdi Zhao and Kurt Keutzer},
  title = {{HallE-Control: Controlling Object Hallucination in Large Multimodal Models}},
  year = {2023},
  eprint = {2310.01779},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.01779},
}
Figure