Domain-Invariant Hallucination Directions for Vision-Language Models

Large vision-language models, which are language models that read images through a visual encoder, often describe objects that are not in the picture. Recent work shows that inside these models, "about to hallucinate" and "grounded" internal states separate along a simple linear direction in the middle layers. Pushing hidden states away from that direction at inference time cuts hallucination substantially (SHARP). Every such result, however, was measured on the same kind of data the direction was learned from, mostly everyday photographs from MSCOCO. In text-only language models, the analogous hallucination detectors collapse to near-chance when moved to a new subject domain, because they latch onto features that identify the domain rather than the error (SpikeScore). Vision-language models degrade badly off the photographic domain: on chest X-rays, accuracy on implicit hallucination probes falls to about 50% (HALLUCINOGEN). Taken together, these findings suggest the hallucination directions we steer with are entangled with image style and domain, and that this entanglement is why they fail where they are most needed.

We propose to learn the hallucination direction the way domain adaptation learns classifiers. The model's middle-layer hidden state is split into a part that predicts "faithful vs. hallucinated" and carries no information about which visual domain the image came from, and a part that absorbs the domain. That second part is prevented from predicting hallucination. The domain-invariant part then serves both as a detector and as a steering vector, and we test it on domains held out entirely from training.

Core Claim

Our claim is that a hallucination steering or detection direction learned with explicit domain disentanglement will transfer to unseen visual domains, while an ordinary one learned on pooled data will not. The unseen domains are stylized photographs, sketches, chest X-rays, and scanned invoices. The claim rests on a premise that nobody has yet tested for vision-language models: in-domain hallucination directions lose most of their benefit under visual domain shift. The first experiment tests that premise directly. If the premise fails, the proposal stops there, and the negative result is itself worth reporting.

Relation to Prior Work

Two papers already bring the word "disentanglement" to multimodal hallucination, and both differ from what we propose. Causal-LLaVA treats object co-occurrence bias (a table triggering hallucinated chairs) as entanglement. It fixes this with a backdoor-adjusted projector and an intervention module built on a predefined dictionary of object categories, and it requires full fine-tuning. It separates objects from other objects, not objects from image domain, and its dictionary cannot represent continuous nuisances such as rendering style or sensor type. Attention Hijackers (AID) disentangles instruction tokens from visual attention at decoding time and raises LLaVA-1.5 POPE Random F1 from 81.75 to 90.12. It works on attention routing rather than on representations, and it is evaluated only in-domain. Neither paper asks whether the fix survives a change of visual domain.

Three papers motivate the premise. SHARP supplies the method we build on: hallucinated states are linearly separable at layers 10–15, and a steering vector injected at layer 10 helps. SpikeScore supplies the warning that activation probes trained on one text domain fall to AUROC 0.5–0.6 on another. Its fix is a behavioral 20-turn pressure test, which is too expensive for decoding-time use and has not been applied to images. We keep the cheap activation-level approach and repair its transfer problem instead. HALLUCINOGEN's medical split and MERLIM's finding that true positives often reflect scene guessing rather than perception (MERLIM) show that the out-of-domain failure is real.

The disentanglement machinery comes from domain adaptation. ESD splits features into invariant and domain-specific parts. It uses a dissimilarity loss between the parts, an adversarial loss so the invariant part cannot predict the domain, and a loss that stops the domain-specific part from predicting the task. Open Compound Domain Adaptation contributes the class-confusion loss on the domain branch. What we change is the target: the "task" here is the model's own grounding failure, read from a frozen model's hidden states, not an image label.

One limitation of that literature constrains our design. Most disentanglement methods assume discrete domain labels. We therefore also test pseudo-domains obtained by clustering visual-encoder features. Approximate Domain Unlearning removes domain information from vision-language models but never measures hallucination. We treat it as evidence that domain information is removable from these models, not as a competitor.

Research Program

The base models are LLaVA-1.5 and LLaVA-Next, kept frozen. We collect the hidden state $h$ at a middle layer $\ell$ (swept over 8–16) at the answer position of POPE-style existence questions. We also collect hidden states at object-noun positions in free captions. Each state is labeled faithful or hallucinated $y$ by checking the model's own answer against ground-truth annotations, and tagged with a visual domain $d$.

Training domains are drawn from a benchmark we build, COCO-Styles. It contains MSCOCO images re-rendered as paintings, cartoons, and sketches by image-to-image translation, with the original object annotations kept. Each rendered image is filtered by requiring an off-the-shelf detector to still find the annotated objects, and a random subset is checked by hand. Held-out domains are one COCO-Styles style at a time, MIMIC-CXR through MED-HALLUCINOGEN, and SROIE invoices using the claim decomposition of ConfLVLM.

A linear encoder maps $h$ into $z_{inv}$ and $z_{spec}$, and a linear decoder reconstructs $h$ from them. Training minimizes

$$\mathcal{L} = \mathcal{L}_{hal}(z_{inv}, y) - \lambda_1 \mathcal{L}_{dom}(z_{inv}, d) - \lambda_2 \mathcal{L}_{hal}(z_{spec}, y) + \lambda_3 \mathcal{L}_{sim}(z_{inv}, z_{spec}) + \lambda_4 \lVert h - \hat h \rVert^2.$$

Here $\mathcal{L}_{hal}$ is a cross-entropy hallucination probe, and $\mathcal{L}_{dom}$ is a domain classifier trained adversarially through gradient reversal. The negative signs mean the encoder tries to make domain unpredictable from $z_{inv}$ and hallucination unpredictable from $z_{spec}$. $\mathcal{L}_{sim}$ is ESD's batch-wise structural-similarity penalty, $\hat h$ is the reconstruction, and the $\lambda$ weights are tuned on a held-in validation domain only.

The probe weight on $z_{inv}$, mapped back through the decoder, gives a hallucination direction $v$ in the original space. We use it two ways. As a detector, we score $h \cdot v$. As a steering vector, we subtract $\alpha v$ at layer $\ell$ during decoding, as SHARP does. For the case without domain labels, $d$ is replaced by k-means clusters of CLIP image embeddings.

Baselines

The naive reference point is the unmodified model with greedy decoding on each held-out domain. Every other number is reported as a change from it.

The strongest representation-level competitor is SHARP, a steering vector derived from contrastive stimuli and injected at layer 10. SHARP reports POPE adversarial accuracy of 82.8% on LLaVA-Next (versus 79.5%) and CHAIR_S of 34.8 (from 52.8) on LLaVA-1.5. We re-derive SHARP's vector from our training domains. We use the same hidden-state pool, layer sweep, and steering-strength search on the same held-in validation domain, so that the only difference is the disentanglement objective.

The strongest training-free competitors are RITUAL and AID. RITUAL averages predictions over randomly transformed images and reports POPE accuracy of 88.87% versus 84.13% on LLaVA-1.5. AID's numbers are given above. Both are run with their published settings on every held-out domain, since neither needs training data. No reported result found on Lacuna for any of these methods on stylized images, X-rays, or invoices, so those cells are new measurements.

Causal-LLaVA is included where its released weights allow. It is the one prior method that also claims disentanglement.

Three ablations isolate our ingredient. The first is a plain linear probe on pooled multi-domain data. The second applies linear domain erasure: project out the directions that predict domain from $h$, then fit the probe. The third is our full objective with $\lambda_1 = \lambda_2 = 0$. Beating these means that on held-out domains, our steering yields a lower POPE-adversarial error and lower CHAIR than each baseline, averaged over seeds with confidence intervals. Detection must also give higher AUROC. In-domain COCO performance must stay within noise of SHARP.

Why not just…?

Why not just train SHARP's vector on pooled data from many domains? This is the most serious threat. Pooling may be enough if the true hallucination direction dominates the domain directions. The pooled-probe ablation settles it directly. If pooling matches our method on held-out domains, we will report that the adversarial split is unnecessary. Our expectation that it will fall short comes from SpikeScore: probes trained on one domain latch onto spectral features specific to that domain. Pooling reduces this but does not forbid it on a domain never seen.

Why not just subtract per-domain means or linearly erase domain information? This is a cheaper version of the same idea, and if it works as well, the right conclusion is "linear erasure suffices." We would accept that result. The difference is that erasure removes every domain-predictive direction, including ones that are also hallucination-predictive, because X-rays genuinely induce more fabrication. Our objective instead forbids hallucination information only in the domain branch. The erasure ablation, together with a measurement of how much in-domain accuracy each method gives up, distinguishes the two.

Why not just use a training-free method like RITUAL or AID, which has no transfer problem? These methods do not learn from domain data, but they still embed photographic assumptions. RITUAL's flips and crops preserve style, and AID's hijacker statistics are calibrated on natural images. Both also add inference cost: RITUAL doubles visual processing, while steering adds one vector addition. Whether they already work on X-rays is unknown (no reported result found on Lacuna). If they do beat us there, the proposal's practical value shrinks to cost, and we will say so. We also test the combination of steering with RITUAL.

Why not just fine-tune on medical or document data? This requires labeled hallucination data in every target domain. Context-Informed Grounding Supervision also shows that forcing heavier reliance on context erodes other knowledge. Our setting assumes the target domain is unseen, so we report per-domain fine-tuning only as an upper reference.

Proposed Experiments and Decisive Checks

The first check tests the premise. We train SHARP-style and plain-probe directions on natural COCO only and evaluate them on each COCO-Styles style, MED-HALLUCINOGEN, and SROIE. The premise holds if the hallucination reduction on held-out domains is a small fraction of the in-domain reduction, and if detection AUROC drops toward 0.5–0.6 as it does in text. If directions transfer well, the project reports that and ends.

The main experiment is leave-one-domain-out training over the five domains. We report POPE accuracy and F1 in the random, popular, and adversarial settings, together with yes-rate. We also report CHAIR_S and CHAIR_I on captions, HALLUCINOGEN affirmative error rates for localization, visual-context, and counterfactual prompts, MERLIM inpainting precision drop, and ConfLVLM-style claim error on SROIE. MME and MM-Vet check that general ability does not regress.

Two diagnostic checks guard against a cosmetic win. First, a fresh domain classifier trained on $z_{inv}$ should sit near chance, while $z_{spec}$ should identify domains easily. Second, steering along $v$ should reduce hallucination on MERLIM's inpainted images, where the object was erased, without lowering recall on the original images. A pure "say no" direction would fail that test by also cutting true positives.

Finally, replacing true domain labels with CLIP-cluster pseudo-domains tests whether the method survives without domain metadata. This is the open-world case that the disentanglement literature leaves open.