Large Vision-Language Models (LVLMs), such as GPT-4V and LLaVA, have demonstrated remarkable capabilities in multimodal reasoning. However, these models frequently suffer from "hallucinations"—instances where the generated text fails to accurately reflect the visual input. This survey provides a comprehensive framework for understanding these failures through the lens of explainability, categorizing misalignment into specific semantic levels and evaluating the strategies designed to mitigate them.
Defining Alignment and Misalignment
The authors distinguish between two primary aspects of alignment that are often conflated in general research:
- Representational Alignment: This refers to the proximity of visual and textual vectors within the model's latent embedding space. If the internal representations are not aligned, the model cannot effectively bridge the gap between modalities.
- Behavioral Alignment: This focuses on the accuracy of the model's final output. Even if representations are somewhat aligned, the model's generation process might still produce errors due to decoding issues or linguistic biases.
To provide a more granular understanding than the broad term "hallucination," the paper proposes a three-level semantic taxonomy for misalignment:
- Object Level ($s_o$): The model fails to detect an object or hallucinates one that is not present in the image.
- Attribute Level ($s_a$): The model correctly identifies an object but misattributes its properties, such as color, size, or material (e.g., calling a green apple "red").
- Relational Level ($s_r$): The model fails to understand the spatial or functional relationships between multiple objects (e.g., claiming a "cat is reading a book").
Root Causes: The Ability Gap and Knowledge Conflict
Why does misalignment occur? The survey identifies several key drivers rooted in model architecture and training data:
- The Ability Gap: LLMs are often significantly more sophisticated than the vision encoders they are paired with. This imbalance leads the model to rely on its internal linguistic priors—what it "thinks" should be there based on text patterns—rather than the specific visual evidence provided in the image.
- The Platonic Representation Hypothesis: While vision and language are different modalities, they are believed to be projections of a shared underlying reality. Alignment is possible because of this common semantic structure, but current architectures often fail to capture it fully during the fusion process.
- Knowledge Conflict: During the fine-tuning phase, task-specific requirements may clash with the general knowledge acquired during pre-training. This can lead to "catastrophic forgetting," particularly regarding complex relational connections.
Mitigation Strategies
The survey categorizes efforts to fix misalignment into two main approaches based on computational intensity:
Parameter-Tuning
These methods involve modifying the model's weights to improve its grounding. Techniques include Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and structural architectural changes to the vision-language connector.
Parameter-Frozen
These are lightweight, training-free strategies that operate during the inference stage. Examples include Retrieval-Augmented Generation (RAG), specialized decoding strategies that penalize hallucination, and post-generation correction loops where the model reviews its own output for consistency.
Benchmarking and Performance Trade-offs
Evaluating alignment requires specialized benchmarks. The paper highlights tools like POPE (for object existence), CHAIR (for captioning hallucinations), and MME (for comprehensive reasoning). A notable trend is the shift from traditional rule-based evaluation toward "Third-model" evaluation, where a more powerful model like GPT-4 is used to grade the outputs of smaller LVLMs.
Comparative analysis on the POPE benchmark reveals a significant trade-off between precision and computational efficiency. For instance, LogicCheckGPT achieves an impressive precision of 98.3% by rigorously verifying its own logic. However, this comes at a high cost: it takes approximately 51.5 seconds per input, compared to just 1.27 seconds for the LLaVa-V1.5-7B baseline. Other methods, like SoM-LLaVA, prioritize "seeing" more details (recall) but may sacrifice some accuracy in the process.
Future Directions
The survey concludes by identifying critical gaps in the current landscape. There is a pressing need for:
- Multimodal Expansion: Moving beyond image-text to include video, audio, and sensor data.
- Unified Evaluation: Creating frameworks that assess object, attribute, and relational alignment simultaneously rather than in isolation.
- Mechanistic Interpretability: Developing techniques to look inside the "black box" and identify the specific neural circuits responsible for misalignment.
- Safety in High-Stakes Domains: Validating these models for use in critical fields like autonomous driving and medical diagnostics, where misalignment can have life-altering consequences.
Summary updated