Large Vision-Language Models (LVLMs) are often described as "Large Language Models with eyes." However, researchers have discovered a fundamental flaw in this integration: when the "brain" (the language model) and the "eyes" (the vision encoder) disagree, the brain almost always wins. This phenomenon, where a model ignores visual evidence in favor of internal statistical patterns, is the primary driver of hallucinations in multimodal AI.
The Linguistic Shortcut
At their core, LVLMs are trained to predict the next token in a sequence. Because they are built on top of powerful language models like Llama or GPT-4, they possess a deep understanding of how words relate to one another. While this makes them fluent, it also makes them "lazy."
As Xuweiyi Chen and colleagues identified in Multi-Object Hallucination in Vision Language Models, models often pay a "multi-object tax." When a scene becomes complex, the model stops looking at the pixels and starts relying on linguistic priors—statistical associations learned during text-only pre-training. For example, if a model identifies a "knife" in an image, its internal probability for the word "fork" spikes. It may then "see" a fork that isn't actually there, simply because the two words frequently co-occur in its training data.
When Common Sense Blinds Perception
This reliance on language patterns creates a "visual neglect" where the model's internal world-knowledge overrides the reality of the image. This is most visible in counter-intuitive scenarios.
In ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense, Kankan Zhou demonstrated that models fail when presented with images that contradict common sense, such as a giant pizza next to a tiny chair. Because the language model "knows" chairs are usually larger than pizzas, it may report a standard scene even when the pixels clearly show the opposite.
Similarly, Yingtian Tang identified Concept Association Bias (CAB) in When are Lemons Purple?. Models often treat images as a "bag of concepts." If an image contains a lemon and the word "purple" appears in the prompt, the model might associate the color with the object based on strong linguistic links to other purple objects (like eggplants), failing to "bind" the correct attributes to the correct visual entities.
The Illusion of Understanding
One of the most insidious aspects of this bias is the "hidden hallucination." As Andrés Villa explored in MERLIM, a model might give the correct answer to a question about an image without actually "seeing" the object.
By using digital inpainting to remove objects from an image, Villa found that models would often continue to report the presence of the removed object. This suggests that high scores on standard benchmarks can be inflated by "lucky guesses" where the model's linguistic expectations happen to align with the image content, masking a lack of true visual grounding.
Forcing the Model to Look
To fix this, researchers are developing methods to "de-bias" the language model and force it to prioritize visual input. The most successful approaches involve Contrastive Decoding.
The intuition behind Visual Contrastive Decoding (VCD), proposed by Sicong Leng, is to compare two different outputs:
- The model's output given a clear image.
- The model's output given a distorted, noisy version of the same image.
When the image is noisy, the model relies almost entirely on its linguistic priors. By subtracting the "noisy" probability distribution from the "clear" one, VCD cancels out the linguistic biases and leaves behind the information that is strictly derived from the visual evidence.
A similar logic applies to Contrastive Region Guidance (CRG) by David Wan, which contrasts the model's response to a full image against its response when a specific region is masked. These training-free methods act as a corrective lens, ensuring that the model's "brain" is actually listening to what its "eyes" are telling it.