Attribute and Relational Hallucinations in Multimodal Models

27 concepts27 papers1 proposal

Most research on hallucinations in Multimodal Large Language Models (MLLMs) focuses on object hallucination: the tendency for a model to claim a "dog" exists in an image when there is only a cat. However, as these models become more sophisticated, a more subtle and pervasive failure mode has emerged. A model might correctly identify every object in a room but completely misrepresent their colors, sizes, and spatial relationships.

This shift from "what is there" to "what it looks like" and "where it is" defines the study of attribute and relational hallucinations.

The Limits of Object-Centric Evaluation

Early benchmarks for visual faithfulness, such as the polling-based POPE framework introduced by Yifan Li et al., were designed to expose statistical biases in how models identify object existence. While effective at catching non-existent objects, these metrics often fail to capture fine-grained unfaithfulness. As Assaf Ben-Kish and Moran Yanuka argue, restricting evaluation to a fixed checklist of objects neglects the "open-vocabulary" nature of real-world descriptions, where errors often lie in the adjectives and prepositions rather than the nouns.

The authors of several recent studies note that a model can achieve high scores on object-level metrics while still generating factually incorrect descriptions. For instance, Shangyu Xing et al. point out that a model might correctly identify a plane but hallucinate that its landing gear is retracted. Similarly, Vlad I Morariu et al. found that even when models successfully identify "context objects" to help localize a target, they frequently fail to resolve fine-grained visual attributes like texture or color within those regions.

Why Attributes and Relations Fail

The root of these hallucinations often lies in a conflict between visual evidence and linguistic priors. Chenxi Wang et al. suggest that as information flows through an MLLM, the internal language model's "common sense" can override actual visual signals in the final layers. If a model sees a birthday cake, its training data might bias it to describe a "dining table" even if the cake is on a stool—a phenomenon Seongheon Park and Yixuan Li describe as a failure of global-local similarity.

Furthermore, the "eyes" of these models—often pretrained encoders like CLIP—may be the primary source of the problem. Yufang Liu et al. demonstrate that these backbones often struggle to distinguish between real and fake descriptions of attributes, providing a shaky foundation for the language layers above them. This is compounded by what Hongliang Wei et al. identify as a length bias: the longer and more detailed a model's description becomes, the more likely it is to "break" and begin hallucinating complex properties and interactions.

Mechanisms for Grounding and Consistency

To move beyond simple object detection, researchers are developing methods to force models to be logically and spatially consistent.

  1. Logical Probing: Junfei Wu et al. proposed LogicCheckGPT, which operates on a simple intuition: if a model claims an object exists, it should be able to describe its attributes consistently. If a model says there is a "red car" but then fails to identify a "red object" when queried, a hallucination is likely present.
  2. Hierarchical Visual Guidance: Rather than looking at an image as a flat set of features, Xianwei Zhuang et al. use a "Coarse-to-Fine View Tree." By using game theory to determine which level of visual detail (from a global view to a tiny crop) is most relevant for each word, they ensure that attributes like color or count are grounded in the correct visual granularity.
  3. Attention Diagnostics: Methods like D-LEAF, developed by Tiancheng Yang, attempt to "localize" hallucinations within the model's own architecture. By identifying which attention heads are failing to focus on the relevant visual features for a specific attribute, the system can dynamically shift focus to heads that are more grounded.
  4. External Verification: Some frameworks, such as MARINE by Linxi Zhao et al., use smaller, specialized vision models to provide a "second opinion" during the decoding process, acting as a filter for attributes that the larger MLLM might be tempted to invent.

The Path Forward

The consensus across recent work—from the ALOHa metric by David Chan et al. to the CAOS framework by Shounak Datta and Dhanasekar Sundararaman—is that the field must move toward "fine-grained" evaluation. As Marcella Cornia et al. argue, traditional word-matching metrics are no longer sufficient for the detailed descriptions generated by modern AI. Ensuring that a model is "faithful" now requires verifying not just the presence of the "dog," but the specific shade of its fur and its exact position relative to the "blue chair."

Key Foundational Work

Modeling Context Between Objects for Referring Expression Understanding (2016) — contributed 1 concept elements to this direction.
Attention Correctness in Neural Image Captioning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attribute_hallucination_82b87929,
  title = {attribute hallucination},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attribute-hallucination/rdr_82b8792947414de09062c4e129ecbe4a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_attribute_hallucination_82b87929_v1,
  title = {attribute hallucination},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attribute-hallucination/rdr_82b8792947414de09062c4e129ecbe4a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake