Vision-language models describe things that are not in the picture. They say "sink" because the photo shows a kitchen counter, not because a sink is visible. The usual explanation is that the language model inside them has learned strong habits about which objects go together, and those habits override weaker evidence from the image. This proposal borrows a trick from text-only work on separating facts from reasoning. If you do not penalize a model for failing to predict something it could have looked up elsewhere, it stops storing that thing in its weights. We apply the same trick to the language prior itself.
During visual instruction tuning, a small, separate "blind" module sees only the text and never the image. It is added to the main model's predictions, but only on words that name visual content: objects, attributes, and relations. The blind module already accounts for whatever those words owe to co-occurrence habit, so the main model is pushed to explain only what the image adds. At test time the blind module is unplugged. The result is a single-pass model whose description of visual content has had the guessable part trained out of it. The prior survives as a separate, inspectable piece that can be dialed back in when guessing is actually wanted.
Core Claim
The claim is that object and attribute hallucination can be reduced at its source, inside the weights, with zero inference overhead. The method moves the language-prior component of visual-content prediction into a detachable module during training, rather than subtracting it at decoding time or suppressing it with preference tuning.
The method comes from Pre-training Limited Memory Language Models with Internal and External Knowledge, called LMLM. LMLM uses an annotator to mark factual values in pretraining text and removes them from the loss when a database lookup supplies them. The model therefore never learns to memorize them: a 382M model matches LLaMA2-7B factual precision, and facts can be deleted by removing a database row. We reuse two pieces of it: the annotator that decides which tokens count as "knowledge," and the principle that withheld gradient means withheld storage. In our version, what gets externalized is not world facts but the co-occurrence prior. The external store is not a database but a frozen text-only expert.
The motivating evidence comes from three papers. Evaluating Object Hallucination in Large Vision-Language Models shows that hallucinated objects track training co-occurrence, with the adversarial split degrading MiniGPT-4 F1 from 78.86 to 71.37. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts from image tokens to previously generated text as responses lengthen. Villa et al. shows that even correct answers are often guessed from context rather than read from pixels. Our blind expert conditions on all preceding text, including the model's own earlier output, so it absorbs both the scene-level co-occurrence prior and the narrative-continuation prior behind long-response drift.
The closest existing work is Knowledge Externalization. It pushes specific concepts (named people, dog breeds) out of a multimodal model's weights into stackable memory tokens, using gradient ascent on the base and descent on the tokens, with the goal of reversible unlearning. It never evaluates general hallucination and requires manual token selection. We externalize a distributional habit rather than named concepts, need no runtime retrieval at all, and target hallucination benchmarks directly.
Decoupling Contrastive Decoding, called DCD, is the closest hallucination method. It trains a separate "negative" image projector on hallucinated responses and subtracts its logits at decoding. That keeps the prior inside the backbone, doubles decoding compute, and fixes the subtraction weight by hand. We move the subtraction into training, so the deployed model runs once.
The general idea of a bias-only branch that is used in training and dropped at test time is known from older classification-style visual question answering. What is new here is fourfold. The branch is applied to autoregressive open-ended generation. It is restricted to annotator-selected visual-content tokens, LMLM-style. It conditions on the model's own generated context. And the detached prior is treated as an editable knowledge module rather than discarded.
Research Program
The base system is LLaVA-1.5 7B, chosen because the file contains the most reported numbers for it. The blind prior expert $\pi$ is a low-rank adapter on the same language backbone, trained on the instruction-tuning responses with all image tokens removed. It is then frozen.
A content annotator marks a token set $C$ for each training response, covering object nouns, attribute words, and relation phrases. It parses each response into entities, attributes, and relations, in the style of the scene-graph parsing used by CutPaste&Find. The main model $\theta$ is then instruction-tuned with
$$ \mathcal{L} = -\sum_{t \in C} \log \mathrm{softmax}\big(z_\theta(t) + \beta\, z_\pi(t)\big)[y_t] \;-\; \sum_{t \notin C} \log \mathrm{softmax}\big(z_\theta(t)\big)[y_t]. $$
The symbols are defined as follows. $z_\theta(t)$ is the main model's logit vector at position $t$, given the image and preceding text. $z_\pi(t)$ is the frozen blind expert's logit vector given only the preceding text. $y_t$ is the target token. $\beta \ge 0$ sets how much of the prior is credited to the blind expert. When the blind expert already assigns high probability to $y_t$, the summed distribution is nearly correct, and the gradient into $\theta$ for that token is small. This is the soft analogue of LMLM's loss masking. At inference $\theta$ runs alone.
Three variants isolate the ingredient. The hard-mask variant drops the loss on content tokens whenever $\pi$'s probability for $y_t$ exceeds a threshold, which is a literal transplant of LMLM. The product variant is the equation above. The scope ablation applies the product to all tokens rather than only to $C$; it tests whether LMLM's "annotate what is knowledge" step matters or whether indiscriminate debiasing damages fluency.
The analysis side asks whether the mechanism did what it claims, beyond whether benchmark numbers moved. For every content token in generated captions, we measure image dependence: the divergence between the main model's next-token distribution under the real image and under a blank image. If the prior was really trained out, image dependence on content tokens should rise relative to standard tuning, while it stays flat on function words.
Second, because the prior now lives in $\pi$, it can be studied as a modular store. We can reattach it at inference with weight $\lambda$ and trace the trade-off between faithfulness and commonsense guessing. We can also check whether $\pi$'s top predictions given scene words reproduce the co-occurrence statistics that POPE's adversarial split was built from.
Baselines
The naive reference point is LLaVA-1.5 tuned with the standard recipe, rerun in our pipeline. For orientation, RITUAL reports 84.13% POPE accuracy for this model, and SHARP reports CHAIRs of 52.8. Reported MME perception for the same base differs between papers: 510.0 in SHARP and 605.00 in ARA. We will therefore compare only against our own reruns, never against copied numbers.
The strongest training-based competitor is DCD. It is the method most explicitly designed to cut hallucination without the general-capability loss of preference tuning. It reports 85.11% adversarial POPE F1, against 83.63% for its baseline and roughly 82% for preference-tuned models, while keeping SEED-Bench within 0.1% of baseline. We will train DCD's projectors on the same base model with the same preference data budget and tune its contrastive weight on the same validation split we use for $\beta$. Beating it means a higher adversarial POPE F1 and SEED-Bench within the same 0.1% band, at half its decoding compute.
The strongest training-free competitors are RITUAL, which reports 88.87% POPE accuracy on LLaVA-1.5, and SHARP, which reports CHAIRs falling from 52.8 to 34.8 at 7.04s per sample against 13.97s for contrastive decoding. Both will be run on our checkpoint with their published settings. Both are also orthogonal to our method, so we will report them stacked on top of it as well as alone.
The ablation that isolates the key ingredient is inference-time blind-logit subtraction. It uses the identical frozen $\pi$ and subtracts $\lambda z_\pi$ from the standard model's logits at decoding, with $\lambda$ tuned on the same budget. This holds the prior estimate fixed and changes only where the subtraction happens. Every model receives identical instruction data, a single tuning sweep of equal size, and matched training steps. Our method's extra cost is the blind adapter's text-only forward pass during training, which we report.
Why not just…?
Why not just subtract the blind model's logits at decoding time? This is the most serious threat, because if it matches us, our contribution shrinks to an efficiency gain. Two reasons suggest it will not fully match. First, contrastive subtraction acts on a model whose weights still encode the prior. It penalizes plausible tokens uniformly, which degrades fluency on content the image does support. Second, it doubles the forward passes, the same overhead that hurts VCD at 13.97s per sample in SHARP's comparison. The matched-$\pi$ ablation above settles the question directly. If inference subtraction ties us on POPE and CHAIR, we will say so and present the method as a way to obtain that effect in one pass.
Why not just fine-tune with preference optimization on hallucinated versus faithful answers? DCD documents that this causes likelihood displacement: a 1–4% drop on SEED-Bench, and adversarial POPE F1 near 82%, slightly below the untuned baseline. Preference tuning tells the model which outputs are bad but not why, so it suppresses confidence broadly. Our loss reduces gradient only on the guessable portion of content tokens and leaves everything else untouched. A preference-tuned run on matched data is included, and the SEED-Bench comparison tests this contrast.
Why not just steer hidden states, as SHARP does? SHARP shows that faithful and hallucinated states are linearly separable in middle layers and that a static vector reduces hallucination cheaply. A static vector pushes all inputs in one direction, though. It cannot tell an image-supported "sink" from a guessed one, whereas our objective makes that distinction token by token during training. We expect the two to stack, and we report the combination. If SHARP alone matches our method, the training-time approach is not worth its cost, and we would conclude that.
Why not just retrieve reference knowledge or verify after generation? ARA and CutPaste&Find attack a different failure: the model's ignorance of what something looks like. ARA retrieves image-caption context and is limited by context length and retrieval noise; CutPaste&Find detects hallucinations after the fact with a chain of detectors. Neither changes why the model guessed. Our method requires no auxiliary vision models and no retrieval at inference, and it is compatible with both.
Proposed Experiments
The primary test is the POPE adversarial split, where negatives are exactly the co-occurring objects the blind expert should absorb. The prediction is a gain over the standard-tuned rerun that is largest on adversarial, smaller on popular, and near zero on random. Gains concentrated on random negatives, or absent from adversarial ones, would falsify the mechanism. We will report the yes-rate alongside F1. Removing a prior could overshoot into a "No" bias, which POPE's original paper shows is how these models fail in the opposite direction.
The second test is CHAIR on captions bucketed by length. If the blind expert really absorbs the narrative-continuation prior, the gap between our model and baseline should widen with response length.
General-capability cost is measured on SEED-Bench, MME, MMStar, and MathVista, with SEED-Bench held to the 0.1% band that DCD achieves. R-Bench probes whether including relation phrases in $C$ helps relation hallucinations, which most existing methods leave untreated.
The mechanistic check is the image-dependence measure on content tokens, compared across standard tuning, our method, and the inference-subtraction ablation. Our method should raise image dependence in the weights; the ablation, by construction, cannot.
Finally, the reattachment sweep over $\lambda$ asks whether the detached prior behaves like a modular knowledge component. We measure how adding it back trades POPE adversarial F1 against MME commonsense-reasoning subtasks, where some guessing from context is legitimately useful.
References
- Pre-training Limited Memory Language Models with Internal and External Knowledge
- Evaluating Object Hallucination in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- Knowledge Externalization: Reversible Unlearning and Modular Retrieval in Multimodal Large Language Models
- Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models
- CutPaste&Find: Efficient Multimodal Hallucination Detector with Visual-aid Knowledge Base
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation