Is the Hallucination in the Word? Separating Label Priors from Visual Evidence with Procedural Scenes

Vision-language models often say an object is present when it is not, especially when the scene "should" contain it: a microwave in a kitchen, or a fifth apple after four apples. One explanation is that the name of the object does the damage. Words like "microwave" carry strong co-occurrence expectations learned from web text, and those expectations can outweigh what the image shows. This note proposes to test that explanation directly. We render 3D scenes in which three things are set independently: how much visual evidence for an object is present, how strongly the surrounding scene suggests it, and how the object is referred to. The object can be called by its ordinary name, described only by its defining attributes ("a cylindrical container with one handle"), or given a made-up name bound to that same definition ("a blicket, which is a cylinder with one handle"). The made-up name keeps the referring role of a label but removes its learned prior, so it acts as the key control.

If hallucination comes from the label, then at fixed visual evidence, answers to ordinary-name queries should track scene context, and answers to made-up-name queries should not. The difference gives a direct, per-model estimate of how much each hallucination comes from the word itself. That estimate also suggests a cheap test-time fix: estimate the label-prior contribution by contrasting the two phrasings and subtract it. We then check whether the fix carries over from rendered scenes to real benchmarks such as POPE, CP-Bench and ROPE.

Core Claim

Our claim is that object hallucination in vision-language models can be split into two measurable parts. One part is prior pressure carried by the label. The other is genuine misperception of the image. The two can be separated cleanly only when visual evidence and scene context are controlled parametrically and the label's meaning is held fixed while its learned associations are removed. Using "semantic label embedding" in this sense means swapping a symbolic or prior-laden label for its definition, as Label Definitions Improve Semantic Role Labeling did for PropBank role tags. We treat that swap as an experimental manipulation, not only as an accuracy trick. For a model, query phrasing $q$, visual evidence level $e$ (fraction of the object's defining attributes rendered) and context congruence $c \in \{0,1\}$, we fit

$$\text{logit}\, P(\text{yes} \mid e, c, q) = \alpha_q + \beta_q\, e + \gamma_q\, c ,$$

where $\beta_q$ is evidence sensitivity and $\gamma_q$ is prior sensitivity. The thesis predicts that $\gamma$ is large for ordinary names and close to zero for made-up names with definitions, while $\beta$ stays the same or rises. A model whose $\gamma$ does not drop under made-up names hallucinates for perceptual reasons rather than lexical ones, which is equally worth knowing.

Where This Sits

Three strands of work motivate the question. Multi-Object Hallucination in Vision Language Models shows that after a run of identical objects, models attend to visual tokens less than 20% of the time and copy the previous label, with accuracy on the odd object "near zero" in AAAAB sequences. HalluSegBench shows that segmentation models hallucinate masks from scene context and that text-only negative questions are easily bypassed. Antidote shows that questions presupposing absent objects drive LLaVA-1.5-7B to 5.7% F1 on CP-Bench. All three point at priors but use real or diffusion-edited images, so evidence and context remain entangled. None of them varies the label while holding its meaning fixed.

Two papers constrain the design. Same Answer, Different Representations reports that degrading visual features reduced POPE false positives by making models conservative. A mitigation that only lowers the "yes" rate will therefore look good on existing benchmarks without improving perception. We guard against this by reporting $\beta$ (evidence sensitivity) alongside false-positive rates. FaithEval finds that larger models more often override counterfactual context (GPT-4o falls to 47.5% on counterfactual science questions). This predicts that $\gamma$ grows with scale, and our scaling sweep tests that directly.

Methods come from two places. The rendering approach follows CLEVR and MATE, which used Blender scenes to isolate reasoning from recognition. MATE deliberately linked objects to arbitrary IDs and never to meaningful labels; we cross both. Definition injection comes from the semantic role labeling paper above, including its warning that overly generic definitions gave no gain. The subtraction step resembles NDAD, which builds a hallucination-leaning distribution by masking attention heads. Ours builds the "negative" distribution from a meaning-preserving rephrasing instead of a heuristic head mask.

Research Program

The scene generator. Bare CLEVR primitives have no lexical priors, so we build a small Blender library of composite objects whose ordinary names carry strong context associations. A mug is a cylinder plus a handle. A lamp is a base, a stem and a shade. A microwave is a box, a window and a dial panel. Each composite has three to five defining attributes that can be rendered, removed or partly scaled, for example handle size from zero to full. This gives the continuous evidence axis $e$. Each scene is placed in a congruent setting (a counter, a sink, a fridge-like box for kitchen items) or an incongruent one, with camera, lighting and object count held fixed. This gives $c$. Absent-object scenes with congruent context reproduce POPE's adversarial split. Presupposing questions ("What color is the mug's handle?") reproduce CP-Bench's trap. AAAAB arrangements with bounding-box prompts reproduce ROPE's momentum setting. All three now come with exact ground truth.

The label conditions. Every query is posed in four phrasings that refer to the same object:

  • the ordinary name;
  • the definition alone;
  • a made-up name defined in the prompt;
  • a made-up name with no definition, the semantically empty label that serves as a floor.

Definitions are written once per object class and checked against the renderer so that each defining attribute matches a rendered part.

Measurement. For each model and phrasing we fit $\beta$ and $\gamma$ from the equation above. We also compute three further quantities:

  • the false-yes rate on absent objects;
  • the presupposition-rejection rate;
  • the odd-object accuracy in momentum sequences.

Following the Same Answer paper, we additionally record final-layer embedding drift between paired congruent and incongruent renders. This tells us whether a lower $\gamma$ under made-up names reflects a real change in internal state or only in output.

Mitigation. The fix, label-prior subtraction, uses two forward passes. One runs the ordinary-name query and the other runs the made-up-name-plus-definition query. The answer distribution is adjusted by removing a scaled share of their log-probability difference, with the scale tuned on rendered scenes only. A second variant simply always asks with the definition. Both are training-free.

Transfer. We freeze everything tuned on rendered data and evaluate on real benchmarks. This addresses the sim-to-real gap that limits procedural benchmarks.

Baselines

The naive reference is the unmodified model (LLaVA-1.5-7B, InstructBLIP and Qwen2.5-VL at 7B and 32B) asked with ordinary names under greedy decoding. Beating it is necessary but tells us little.

The strongest training-based competitor is Antidote. It generates synthetic images and presupposition questions, then trains with preference optimization so the model prefers self-corrected answers. The paper reports LLaVA-1.5-7B rising from 5.7% to 78.4% F1 on CP-Bench and gains of 1.8–3.3% on POPE. We run Antidote from released weights if available. Otherwise we re-implement it on LLaVA-1.5-7B with its published data recipe and compare at equal evaluation conditions. Antidote uses training, which we do not, so matching its CP-Bench F1 within a few points without training would count as a strong result. We also report Antidote's own $\gamma$ and $\beta$ on our renders, which tests whether its gains come from reduced prior pressure or from a general lean toward "no".

The strongest test-time competitor is the causal intervention of Treble Counterfactual VLMs. It suppresses direct vision-only and text-only effects in intermediate representations and is reported to improve LLaVA-1.5 POPE F1 by "over 10%". We apply it with the authors' setup on the same models. Our subtraction method gets the same budget of calibration examples, roughly 50 as in that paper. To count as beating it, we must match or exceed its POPE F1 gain while also keeping evidence sensitivity $\beta$ at least as high on renders. NDAD is run as a second test-time baseline on the language backbone, at its reported cost of about 20% extra runtime. Our two-pass method costs about 100% extra, and we report this honestly.

The key ablations isolate the ingredient. Made-up name without definition tests whether removing the prior alone suffices, which would mean the definition contributes nothing. Ordinary name plus definition tests whether adding meaning helps when the prior is still present. A generic definition and a length-matched irrelevant sentence test whether any gain comes simply from extra prompt tokens, given the generic-definition null result in the semantic role labeling paper.

Why not just…?

Why not just use a bigger model? Scale is the hypothesis under test, not an answer to it. FaithEval predicts that stronger priors grow with size. MATE shows Claude 3.5 Sonnet at 83.3% against 97.9% for humans even on prior-free synthetic scenes. Our Qwen2.5-VL 7B-versus-32B comparison reports $\gamma$ directly. If $\gamma$ shrinks with scale, this objection wins on that axis, and we will say so.

Why not just ask the model to describe the object's attributes first, chain-of-thought style? This is the real threat. A self-generated definition might do everything an external one does. The file reports that chain-of-thought helps some models and hurts others on MATE and Isle, so the outcome is genuinely open. We therefore include a condition in which the model writes its own definition before answering, using the same token budget. If it matches external definitions, the contribution shifts from the method to the diagnosis: models already hold the meaning but need to be made to use it instead of the name.

Why not just prompt the model to say "no" when unsure? Abstention prompts lower false positives by lowering all "yes" answers. The Same Answer paper shows this conservatism can pass for robustness on POPE. Because we report the evidence slope $\beta$, a model that simply says "no" more often will show a lower $\beta$, which separates it from one that has stopped listening to the label.

Why not just fine-tune on counterfactuals, as Antidote and RobustSeg do? Fine-tuning works on its benchmarks, and we include it as a baseline. It does not say why the model hallucinated, and it requires training for every model. If Antidote lowers $\gamma$ as much as our method does, we gain a mechanistic account of why it works. If it lowers false positives without lowering $\gamma$, we gain evidence that it learned a response bias.

Proposed Experiments and Decisive Checks

The first decisive check runs on rendered scenes. At matched evidence $e$, ordinary-name queries must show significantly larger prior sensitivity $\gamma$ than made-up-name-plus-definition queries, with bootstrap confidence intervals over scenes. If the two are equal across models, hallucination here is not carried by the label and the subtraction method has no basis.

The second check is presupposition and momentum. On rendered presupposing questions and AAAAB sequences, definition phrasing should raise rejection rates and odd-object accuracy. We will also measure whether attention to visual tokens rises above the sub-20% level reported for ROPE.

The third check is transfer with frozen settings to:

  • POPE (all three splits, F1);
  • CP-Bench (F1, against Antidote's reported 78.4%);
  • CHAIR (no comparable number for our method yet);
  • ROPE adversarial sequences.

The fourth check is scaling. We plot $\gamma$ and $\beta$ against model size to see whether larger models are more prior-driven (FaithEval's direction) or more perceptually fragile (the Same Answer direction). This settles, for object existence, which of the two failure modes dominates.