Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability

Shuwen Dong, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Cheng Lü, Mengnan Du2025 · DOI

Large Vision-Language Models (LVLMs), such as GPT-4V and LLaVA, have demonstrated remarkable capabilities in multimodal reasoning. However, these models frequently suffer from "hallucinations"—instances where the generated text fails to accurately reflect the visual input. This survey provides a comprehensive framework for understanding these failures through the lens of explainability, categorizing misalignment into specific semantic levels and evaluating the strategies designed to mitigate them.

Defining Alignment and Misalignment

The authors distinguish between two primary aspects of alignment that are often conflated in general research:

  1. Representational Alignment: This refers to the proximity of visual and textual vectors within the model's latent embedding space. If the internal representations are not aligned, the model cannot effectively bridge the gap between modalities.
  2. Behavioral Alignment: This focuses on the accuracy of the model's final output. Even if representations are somewhat aligned, the model's generation process might still produce errors due to decoding issues or linguistic biases.

To provide a more granular understanding than the broad term "hallucination," the paper proposes a three-level semantic taxonomy for misalignment:

  • Object Level ($s_o$): The model fails to detect an object or hallucinates one that is not present in the image.
  • Attribute Level ($s_a$): The model correctly identifies an object but misattributes its properties, such as color, size, or material (e.g., calling a green apple "red").
  • Relational Level ($s_r$): The model fails to understand the spatial or functional relationships between multiple objects (e.g., claiming a "cat is reading a book").

Root Causes: The Ability Gap and Knowledge Conflict

Why does misalignment occur? The survey identifies several key drivers rooted in model architecture and training data:

  • The Ability Gap: LLMs are often significantly more sophisticated than the vision encoders they are paired with. This imbalance leads the model to rely on its internal linguistic priors—what it "thinks" should be there based on text patterns—rather than the specific visual evidence provided in the image.
  • The Platonic Representation Hypothesis: While vision and language are different modalities, they are believed to be projections of a shared underlying reality. Alignment is possible because of this common semantic structure, but current architectures often fail to capture it fully during the fusion process.
  • Knowledge Conflict: During the fine-tuning phase, task-specific requirements may clash with the general knowledge acquired during pre-training. This can lead to "catastrophic forgetting," particularly regarding complex relational connections.

Mitigation Strategies

The survey categorizes efforts to fix misalignment into two main approaches based on computational intensity:

Parameter-Tuning

These methods involve modifying the model's weights to improve its grounding. Techniques include Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and structural architectural changes to the vision-language connector.

Parameter-Frozen

These are lightweight, training-free strategies that operate during the inference stage. Examples include Retrieval-Augmented Generation (RAG), specialized decoding strategies that penalize hallucination, and post-generation correction loops where the model reviews its own output for consistency.

Benchmarking and Performance Trade-offs

Evaluating alignment requires specialized benchmarks. The paper highlights tools like POPE (for object existence), CHAIR (for captioning hallucinations), and MME (for comprehensive reasoning). A notable trend is the shift from traditional rule-based evaluation toward "Third-model" evaluation, where a more powerful model like GPT-4 is used to grade the outputs of smaller LVLMs.

Comparative analysis on the POPE benchmark reveals a significant trade-off between precision and computational efficiency. For instance, LogicCheckGPT achieves an impressive precision of 98.3% by rigorously verifying its own logic. However, this comes at a high cost: it takes approximately 51.5 seconds per input, compared to just 1.27 seconds for the LLaVa-V1.5-7B baseline. Other methods, like SoM-LLaVA, prioritize "seeing" more details (recall) but may sacrifice some accuracy in the process.

Future Directions

The survey concludes by identifying critical gaps in the current landscape. There is a pressing need for:

  • Multimodal Expansion: Moving beyond image-text to include video, audio, and sensor data.
  • Unified Evaluation: Creating frameworks that assess object, attribute, and relational alignment simultaneously rather than in isolation.
  • Mechanistic Interpretability: Developing techniques to look inside the "black box" and identify the specific neural circuits responsible for misalignment.
  • Safety in High-Stakes Domains: Validating these models for use in critical fields like autonomous driving and medical diagnostics, where misalignment can have life-altering consequences.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in processing both visual and textual information. However, the critical challenge of alignment between visual and textual representations is<br/>not fully understood. This survey presents a comprehensive examination of alignment and misalignment in LVLMs through an explainability lens. We first examine the fundamentals of alignment, exploring its representational and behavioral aspects, training methodologies, and theoretical foundations. We then analyze misalignment phenomena across three semantic levels: object, attribute, and relational misalignment. Our investigation reveals that misalignment emerges from challenges at multiple levels: the data level, the model level, and<br/>the inference level. We provide a comprehensive review of existing mitigation strategies, categorizing them into parameter-frozen and parameter-tuning approaches. Finally, we outline promising future research directions, emphasizing the need for standardized evaluation protocols and in-depth explainability studies.
BibTeX
@misc{dong2025large,
  author = {Shuwen Dong and Haiyan Zhao and Jingyu Hu and Weiru Liu and Ali Payani and Cheng Lü and Mengnan Du},
  title = {{Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.90},
}
Figure