When a vision-language model writes a long description of an image, the errors usually do not start at the first sentence. They build up later: after "a kitchen with a dining table," the model adds "forks and wine glasses" that are not in the picture, because its own earlier words make those objects likely. This proposal builds a cheap check that runs while the model is writing. Before decoding begins, it turns the image into a small structured inventory, called a scene representation, made of object "slots" (groups of image patches that belong to one object) and the pairwise relations between them. Then, at every word that names an object, attribute, or relation, it measures two things from the model's attention. The first is how much the word draws on the matching slot. The second is how much it draws on distant earlier text. A word that looks away from the image and leans on remote text is flagged, and decoding is steered back toward the slots before the word is emitted.
The opening claim is that hallucinations in long outputs are a long-range context failure: the model fills in later content from distant text rather than from the image. A detector that asks "which object in this image supports this word, and how far back in the text is the word actually reaching?" should therefore beat detectors that read a single unstructured hidden state. The advantage should be largest on late-position words and on relation claims, where object-existence checks say nothing.
What exists and where the gap is
The file supports the real-time half of this well. Simple linear classifiers on intermediate hidden states detect ungrounded claims (Do Androids Know They're Only Dreaming of Electric Sheep?). Semantic entropy, a measure of how much a model's possible answers disagree in meaning, can be approximated by a linear probe without repeated sampling (Semantic Entropy Probes). Prompt-time hidden states predict upcoming factual errors "with over 70% accuracy" (FactCheckmate). Several papers characterize the internal geometry of untruthful generation: local intrinsic dimension of the activations (Local Intrinsic Dimension), contributions to the residual stream, which is the running sum each layer adds to (ICR Probe), layer-to-layer state transitions modeled with hidden Markov models (PoLLMgraph), and attention sinks (Attention Sinks as Internal Signals).
These papers motivate the proposal. They show that internal signals are informative and cheap to read. They do not supply its method, because each reads an unstructured vector and has no notion of which object in the image a word should rest on.
The closest multimodal work is Instruction Lens Score. It projects hidden states into vocabulary space and computes a calibrated local score and a context-consistency score for object hallucination. It reports outperforming GLSIM by 7.70% AUROC and 3.29% AUPR on MSCOCO with Qwen3-VL, at about 100 ms of latency. This is the strongest competitor and the reference latency budget. It scores object tokens against image evidence. It does not decompose evidence into object slots or relations, and it does not measure how far back in the text a prediction reaches. Those two additions are what this proposal tests.
The research file was truncated before its section on existing work at this intersection. It contains no papers on structured scene representation or on long-range dependency modeling. Whether a slot-conditioned, distance-aware decoding monitor already exists is therefore unverified here, and the scene-representation and long-range components are built from standard techniques rather than from cited prior work.
Core Claim
Define $a_t^{\text{slot}}$ as the attention mass that word $t$ places on the image tokens of the slot matched to the word. The match comes from comparing the word's embedding with each slot's pooled embedding. Define $a_t^{\text{far}}$ as the attention mass on text tokens more than $d$ positions back. The grounding-gap score is
$$s_t = \sigma\big(w^\top h_t\big) + \lambda\,\big(a_t^{\text{far}} - a_t^{\text{slot}}\big),$$
where $h_t$ is the hidden state at a chosen layer, $w$ is a linear probe of the kind in Do Androids Know, $\sigma$ is the logistic function, and $\lambda$ is a weight fitted on held-out data. For relation words such as "on," "holding," or "next to," $a_t^{\text{slot}}$ is replaced by attention on the union of the two slots that the relation joins.
The prediction is specific. Adding the structured, distance-aware term should help most where an unstructured probe and the instruction-lens score are weakest: words late in long descriptions, and relation or attribute claims.
Research Program
Slots are built once, during prefill, so per-token cost stays near that of reading attention weights. Two slot builders will be compared. The first clusters the model's own visual tokens by feature similarity. The second assigns patches to boxes from an off-the-shelf open-vocabulary detector. The comparison shows whether the gain comes from the model's internal structure or from external perception.
Relations between slots are represented minimally, as slot pairs whose patches are spatially adjacent or overlapping. The aim is not to recover a full scene graph. It is to give relation words a place to look.
Mitigation reuses the score. When $s_t$ crosses a threshold at an object or relation word, the model re-decodes that one step. Attention logits on far-back text are scaled down, and attention on candidate slots is boosted. Alternatively, the model is allowed to emit a hedge or skip the claim. This intervention touches a single step and does not roll back the sequence, which keeps latency bounded.
Baselines
The naive reference is ordinary greedy decoding of Qwen3-VL with no monitor. It sets the hallucination rate and object recall that every intervention must improve without cutting recall heavily.
The first internal-signal baseline is a linear hidden-state probe as in Do Androids Know, alongside Semantic Entropy Probes. Both are trained on the same labeled tokens, the same layers, and the same hyperparameter search budget as our probe term $w$. Our score contains this probe, so any gain must come from the added structured term.
The strongest existing method is Instruction Lens Score. The file gives only its relative result: 7.70% AUROC and 3.29% AUPR above GLSIM on MSCOCO with Qwen3-VL, at about 100 ms. It reports no absolute AUROC that could serve as a target number. We will re-run it on our exact splits and model, with its calibration fit on the same data as ours. Beating it means higher token-level AUROC and AUPR at comparable or lower added latency, with confidence intervals from bootstrap over images.
A detector-matching baseline flags any named object absent from the open-vocabulary detector's output. It uses the same detector as our second slot builder.
Two ablations isolate the key ingredients. The first replaces slot attention with total attention on all image tokens, which removes structure but keeps the long-range term. The second drops $a_t^{\text{far}}$, which keeps structure but removes the long-range term. The proposal's claim survives only if the full score beats both ablations on late-position and relation words.
Why not just…?
Why not just train a better linear probe on hidden states? Probes already work well (Do Androids Know, FactCheckmate), and a hidden state may already encode slot and distance information implicitly. This is a real threat. The probe-only baseline settles it. Results are reported split by word position and by claim type. If the probe matches the full score on late relation words, the structured term is unnecessary, and we would report that.
Why not just check named objects against a detector? For object existence, this may genuinely tie or win, since it consults independent perception. The proposal does not claim otherwise. Its claim rests on relations and attributes, where a list of detected objects cannot say whether "the cat is on the sofa" is true. It also rests on cases where the detector misses small objects that the model did correctly attend to. If detector matching wins on objects, the paper's scope narrows to relations.
Why not just penalize attention to distant text, with no slots? Decoding-time methods described in the file (for example, retrospection-and-rollback approaches) already target over-reliance on earlier text. The first ablation is exactly this alternative. If it captures most of the gain, the slot structure is ornamental and the contribution reduces to the long-range term.
Proposed Experiments
Detection is evaluated on MSCOCO images, following the setting of Instruction Lens Score, with Qwen3-VL generating long descriptions. Object-word labels come from matching generated nouns against COCO object annotations. Relation and attribute labels require new annotation. We will hand-label relation claims in a sample of a few hundred descriptions, because the file names no relation-hallucination benchmark. The metrics are token-level AUROC and AUPR. Results are reported overall, by position bucket (for example, first versus later thirds of each description), and by claim type. The added milliseconds per generated token are measured on the same hardware for every method.
Mitigation is evaluated by the rate of hallucinated objects and relations per description, by object recall relative to annotations, and by end-to-end latency.
The decisive check is the interaction pattern. The idea is supported if the full score's advantage over both Instruction Lens Score and the probe-only baseline grows with word position and is largest on relation claims, while both ablations fall short there. If the advantage is flat across positions, the long-range story is wrong. If the slot-free ablation matches the full score, the structure story is wrong. Either outcome would be reported as the finding.
References
- Do Androids Know They're Only Dreaming of Electric Sheep?
- Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs
- FactCheckmate: Preemptively Detecting and Mitigating Hallucinations in LMs
- Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension
- ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs
- PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics
- Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
- Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models