Large Vision-Language Models (LVLMs) often suffer from a "hallucination" problem: they describe objects, attributes, or relationships that do not exist in the input image. While early explanations attributed this to poor training data, recent research suggests a more fundamental structural cause. Hallucinations emerge from a tug-of-war between visual evidence and the model’s internal language priors—a phenomenon often called "textual inertia."
To solve this, researchers are moving beyond simple retraining. Instead, they are developing techniques to diagnose and steer the model’s internal attention and decoding processes in real-time, ensuring that the "visual voice" wins out over the "linguistic guess."
The Tug-of-War Between Vision and Language
In a standard transformer-based LVLM, the model processes visual tokens and text tokens in a shared space. As the model generates a response, it must balance two forces: the visual features extracted from the image and the statistical likelihood of the next word based on the text it has already written.
Wenliang Dai and Zihan Liu demonstrated in Plausible May Not Be Faithful that models often prioritize "plausibility" over "faithfulness." For example, if a model sees a kitchen, its language prior might strongly suggest a "toaster" is present, even if the visual tokens don't support it. This over-reliance on internal knowledge rather than visual evidence is the primary driver of object hallucination.
Locating the Engine Room of Hallucination
Where exactly does this breakdown happen? Zhangqi Jiang and colleagues identified that the "middle layers" of the transformer stack act as the critical transition point in Devils in Middle Layers of Large Vision-Language Models. They observed a two-stage process:
- Enrichment: Early layers process raw visual features.
- Refinement: Middle layers begin to integrate these features with linguistic context.
When a model hallucinations, the attention heads in these middle layers stop looking at the image tokens and start focusing almost exclusively on the previously generated text. Nhat Hoang-Xuan and Minh Vu formalized this observation with the Prelim Attention Score (PAS) in PAS : Prelim Attention Score for Detecting Object Hallucinations, showing that internal attention weights can serve as a real-time "lie detector" for hallucinations.
Steering the Model at Inference Time
Because hallucinations are rooted in how the model allocates attention during decoding, many researchers have turned to "training-free" interventions that steer the model's output without requiring expensive retraining.
Contrastive Decoding
One powerful approach is to compare the model's "biased" state with its "grounded" state. Jinhe Bi proposed ASCD (Attention-Steerable Contrastive Decoding) in ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM, which amplifies the influence of visual tokens while dampening heads that are overly focused on text. Similarly, Jingran Su and Jingfan Chen developed Activation Steering Decoding (ASD) to intervene directly in the hidden states, neutralizing neural patterns associated with hallucination Activation Steering Decoding.
Dynamic Correction
Instead of just steering, some methods "rescue" visual information that was lost in the final layers. Chenxi Wang and Xiang Chen introduced DeCo (Dynamic Correction Decoding) in MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation, which uses visual information from intermediate layers to correct the final output distribution, effectively forcing the model to "double-check" its work against the image.
Beyond Existence: The Challenge of Attributes and Relations
While current research has become highly effective at preventing models from imagining non-existent objects, a more subtle challenge remains: attribute and relational hallucination.
As noted by Chenxi Liu in Attention Correctness in Neural Image Captioning, a model might correctly identify a "dog" but hallucinate its color (attribute) or its position relative to a "chair" (spatial relationship). Junfei Wu and Qiang Liu address this in Logical Closed Loop by checking for logical consistency: if a model claims an object is "red," it should be able to find that same "red object" when queried again.
The field is now shifting toward these fine-grained errors. Metrics like FAITHSCORE, developed by Liqiang Jing in FAITHSCORE: Evaluating Hallucinations in Large Vision-Language Models, decompose descriptions into atomic facts to verify not just that the objects exist, but that their properties and interactions are grounded in reality.