The word "old" looks different on a tiger, a car and a street. Graying fur, rust and cracked asphalt have almost nothing in common at the pixel level. Large vision-language models (LVLMs), which are image encoders attached to a text-generating language model, are nonetheless asked to confirm or describe "old," "wet" or "sliced" with one shared internal pathway. We conjecture that a substantial share of their attribute hallucinations are context-transfer errors. The model has learned what an attribute looks like mostly on the objects it usually appears with. When the attribute shows up on an unusual object, the model either misses it or asserts it from language habit.
This note proposes two things. The first is a test of that conjecture. The second is a targeted fix: a mixture of experts, meaning a layer made of several parallel sub-networks plus a small gating model that picks which ones process each token. Our version differs from standard designs in what the gate looks at. It is keyed on the image region the model is attending to while it produces or checks an attribute word, not on the token's own hidden state alone. As a result, "old on animals" and "old on vehicles" can be computed by different parameters.
Core Claim
Hallucination research has mostly measured whether objects exist. Evaluating Object Hallucination in Large Vision-Language Models built POPE, a set of yes/no existence probes, and traced errors to object co-occurrence statistics. Work on attributes has stayed diagnostic. Evaluating Attribute Comprehension in Large Vision-Language Models showed that LVLMs treat attributes like a bag of words on VAW, and it proposed no remedy.
Compositional zero-shot learning is the task of recognizing unseen attribute–object pairs, and its literature supplies the missing theory. From Red Wine to Red Tomato: Composition with Context and Attributes as Operators argue that an attribute's appearance is a function of its object. Learning Unseen Concepts via Hierarchical Decomposition and Composition shows that a single static prototype for an attribute fails when it is moved to a new object family.
Our claim has two parts, and both can be falsified. The first part is that LVLM attribute errors grow with context displacement: how far an object sits from the objects an attribute usually co-occurs with in training text, holding attribute frequency fixed. The second part is that routing attribute computation by visual object context reduces exactly these displaced errors. The test is a comparison against a dense model and a token-routed mixture of experts that have the same number of parameters and see the same training data. The gain should concentrate on high-displacement pairs and should not appear as a uniform shift.
Relation to the Closest Work
Two papers bracket the idea.
Decoupling Primitive with Experts (EVA) inserts LoRA-based expert adapters into a frozen CLIP model; LoRA adds small low-rank weight updates to a frozen network. EVA's experts include one shared expert and others specialized by facet, such as color or texture. EVA addresses attributes with many visual meanings, but only for classification on MIT-States, UT-Zappos and C-GQA. It never generates text and never measures hallucination. EVA also specializes experts by attribute facet, whereas we specialize them by object context. These are different axes: a color expert still has to handle "red wine" and "red tomato" with the same parameters.
MoE-LLaVA supplies the method we reuse. It upcycles an LVLM into sparse experts by copying the dense feed-forward weights into each expert and then training a top-2 gate per token, and it reports that this beats a dense 13B model on POPE. Its gate is generic over tokens, and it never evaluates attributes.
Other relevant work frames the problem rather than supplying a method:
- CSCNET and CANet condition attribute classifiers on a predicted object. They motivate keying on the object, but they are classifiers, not generators.
- MoLE calls fixed transformer layers "experts" at decoding time. It has no specialized parameters.
- Discovering Compositional Hallucinations in LVLMs studies compositional failures at the level of image-plus-knowledge reasoning, not attribute appearance. Its "blank image degrades text reasoning" finding constrains our design: adding visual signal to the gate could itself interfere with language computation. This is why we keep a shared expert and measure text-only regressions.
Research Program
Diagnostic. We first build a probe set from datasets whose labels tie attributes to objects. MIT-States and C-GQA contribute state–object pairs. VAW contributes fine-grained attributes. PACO contributes attributes annotated per object and per part, such as "chair-leg" versus "dog-leg."
From these we generate POPE-style yes/no questions ("Is the tiger old?") and negatives in which the attribute fits a co-occurring object but not this one. For each attribute–object pair we compute displacement as the text-embedding distance between the object and the attribute's most frequent co-occurring objects in the LVLM's instruction-tuning captions.
We then regress the false-yes and false-no rates on displacement, controlling for attribute frequency and object frequency. We run this on LLaVA-1.5 and on MoE-LLaVA.
Mechanism. We upcycle LoRA experts in the language model's feed-forward layers, following MoE-LLaVA's staged training, with one always-on shared expert in the style of EVA. The gate for each token receives two inputs. One is the token's hidden state. The other is the visual-attention context: the attention-weighted mean of the image tokens at that decoding step, summarizing "what the model is looking at now."
This choice handles an ordering problem. English places adjectives before nouns, so a gate keyed on the noun text could not see the object before emitting "red." The image region, by contrast, is already being attended to at that step.
We optionally add an auxiliary loss that encourages routing to cluster by object supercategory (for example animals versus vehicles). We report results with and without it.
Baselines
Naive reference points. The do-nothing reference is the unmodified LLaVA-1.5 evaluated on our probes. We found no reported result on Lacuna for LVLMs on displacement-stratified attribute probes, so we will establish this baseline ourselves. A second reference is LLaVA-1.5 prompted to name the object before judging the attribute. This is a prompting version of CSCNET's cascade.
Strongest existing hallucination mitigation. This is SHARP, which steers middle-layer activations along learned "hallucination vectors." On LLaVA-1.5 it reports raising the MME perception total from 510.0 to 613.3 and lowering sentence-level CHAIR from 52.8 to 34.8. On LLaVA-Next it reports adversarial POPE accuracy rising from 79.5% to 82.8%. We fit SHARP's vectors on the same training split of our probes and tune its layer and strength on the same validation split.
Training-free decoding baselines. We run RITUAL, which reports LLaVA-1.5 POPE accuracy rising from 84.13% to 88.87%, and MoLE, both at their published settings.
Matched training baselines. These isolate our key ingredient and are the most important comparisons. All three are trained on identical data with matched active parameters, steps and learning-rate sweeps:
- a dense LoRA fine-tune;
- a MoE-LLaVA-style token-routed mixture;
- our object-conditioned mixture.
We also include a control that tests whether the specific visual input to the gate matters. It feeds the gate a random image region, or the whole-image mean, instead of the attended region.
What counts as success. Beating the baselines means a statistically reliable reduction in attribute error on the top displacement tercile against every matched baseline. It also requires no loss on POPE, on the MME-hall object subset, or on text-only ScienceQA. A uniform gain spread across all terciles would not support the mechanism claim.
Why not just…?
Why not just fine-tune a dense model on attribute-dense captions? This is the most serious threat. Evaluating Attribute Comprehension in Large Vision-Language Models found that attribute density in captions, not image resolution, drives attribute comprehension. Our matched dense LoRA arm trains on exactly this kind of data. If it equals the object-routed model on high-displacement pairs, we will conclude that the problem is data coverage and drop the architectural claim.
Why not just learn one steering vector per object category? This is a cheaper rival that we take seriously. SHARP uses dataset-level vectors, which cannot encode a correction that flips sign across objects. A family of steering vectors, one per object cluster, could. We therefore add this object-conditioned version of SHARP as an extra baseline. If it matches us, the gating machinery is unnecessary.
Why not just use contrastive decoding with image perturbations? Flips, crops and color jitter, as used in RITUAL, do not change which object an attribute sits on. They also run the model twice. We expect RITUAL's gains to be flat across displacement terciles, and the stratified analysis tests this directly.
Why not just prompt the model to identify the object first? This baseline costs nothing. However, Multi-Object Hallucination in Vision Language Models shows that even explicit visual prompts such as drawn bounding boxes leave models putting under 20% of their attention on visual tokens while they hallucinate. Naming the object may therefore not change how the attribute evidence is computed. The prompted-cascade arm measures this directly.
Proposed Experiments and Decisive Checks
The first decisive check is the diagnostic itself. If attribute error does not rise with displacement after the frequency controls, the premise fails and we stop before any training.
The second check is the matched three-way comparison of dense, token-routed and object-routed models, stratified by displacement. We report it on our probes, on the MME-hall attribute subset, and on PACO part-attribute questions.
The third check is on free-form captioning. We extend CHAIR, the caption hallucination metric based on image object annotations, to count attribute–object pairs against VAW and PACO annotations. We also run the CAOS analysis from Evaluating Hallucination in LVLMs based on Context-Aware Object Similarities, which asks whether remaining errors come from the model's own earlier text.
The fourth check covers regressions. We report POPE (random, popular and adversarial), HallusionBench question-pair accuracy (open-source models reportedly fall below 16%), and text-only ScienceQA, to catch the blank-canvas interference effect.
The fifth check is a routing analysis that asks whether "old on animals" and "old on vehicles" in fact land on different experts. The prediction is that the gate separates them, and that the token-routed mixture's gate does not.
References
- Evaluating Object Hallucination in Large Vision-Language Models
- Evaluating Attribute Comprehension in Large Vision-Language Models
- From Red Wine to Red Tomato: Composition with Context
- Attributes as Operators: Factorizing Unseen Attribute-Object Compositions
- Learning Unseen Concepts via Hierarchical Decomposition and Composition
- Decoupling Primitive with Experts: Dynamic Feature Alignment for Compositional Zero-Shot Learning
- MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
- CSCNet: Class-Specified Cascaded Network for Compositional Zero-Shot Learning
- Learning Conditional Attributes for Compositional Zero-Shot Learning
- MoLE: Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models
- Discovering Compositional Hallucinations in LVLMs
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Multi-Object Hallucination in Vision Language Models
- Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities