Lexical Provenance of Context-Induced Hallucinations in Vision-Language Models

Vision-language models, which read an image and write text about it, are increasingly run with large amounts of extra text in the prompt. That text includes retrieved web passages, chat history, and document clutter. When such a model says something about the image that is not true, the usual explanation is that the language model's habits (it "knows" kitchens have sinks) overpower weak visual evidence. We propose a narrower and more testable explanation for hallucinations that appear only when noisy text is present. The model is often copying: the false word or phrase can be traced, string for string, to the distracting text it was given. If this holds, the crude tool everyone criticizes for evaluation, n-gram overlap (counting shared word sequences), becomes a cheap and precise diagnostic. Overlap between the output and the distractor text is evidence that a claim came from the prompt rather than from the pixels.

The note makes three contributions. The first is a controlled stress test that separates two properties of a distractor passage: whether it shares surface words with the question, and whether it is topically related to the image. The second is a provenance-based detector and a decoding correction that intervenes only on tokens that complete a word sequence copied from the context. The third is a measurement showing that standard overlap metrics reward these copied hallucinations when distractor vocabulary overlaps the reference captions. The two-sentence version: under textual noise, vision-language hallucinations are disproportionately lexical copies of the noise. You can detect and suppress them by checking where the words came from, at lower cost than the internal-state methods now used.

Core Claim

We make one central empirical prediction. When a passage is prepended to a visual question, hallucination rates depend more on whether the passage literally contains the probed word than on whether the passage is semantically about the scene. Moreover, a passage that mentions an object only to deny it ("there is no dog in the yard") raises the false-"yes" rate above a passage that never mentions it. That second effect would be the Keyword Illusion, documented for overlap metrics in Assessing The Factual Accuracy of Generated Text, appearing inside the model. Text-only evidence points this way: DAS found zero-shot readers are pulled toward surface matches between query words and irrelevant documents. Nobody has tested whether the same surface pull dominates when an image is available as an alternative source of truth.

The method follows from the claim. We flag a candidate token as context-sourced if emitting it would complete an n-gram (length $n \ge 2$, or a single content noun) that occurs in the injected context. Only at flagged positions do we compare against an image-free branch. With $\ell_{\text{full}}(y)$ the next-token logit given image, context and prefix, and $\ell_{\text{text}}(y)$ the logit given context and prefix without the image, the corrected logit is

$$\ell'(y) = \ell_{\text{full}}(y) - \alpha\, g(y)\,\big[\ell_{\text{text}}(y) - \ell_{\text{full}}(y)\big]_+ ,$$

where $g(y)\in\{0,1\}$ is the provenance flag, $\alpha$ is a strength parameter, and $[\cdot]_+$ keeps only cases where the image makes the token less likely than text alone would. In words: a copied token is penalized only when the image actively argues against it. The image-free branch contains no visual tokens (hundreds per image for LLaVA-style models), so maintaining its cache is much cheaper than the second full pass required by contrastive decoding.

How This Sits Against Prior Work

Three papers motivate the proposal by showing that context drives multimodal hallucination, but each studies a different kind of context. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that a model's own generated text pulls attention off visual tokens (a "contextual trap"), and it detects this with attention-similarity signals (HalTrapper). It explicitly leaves external context unexamined, and that is our regime. MM-Snowball injects a false premise at turn 3 of a dialogue and observes accuracy below 2% for LLaVA-1.5-7B in turns 3–5. It attributes this to "visual fading" and fixes it with CAVR, which, among other things, contrasts full-context logits with a history-free visual stream. Dyn-VQA/OmniSearch shows retrieved web noise induces "Thinking Traps" in multimodal retrieval agents but offers no decoding-level fix.

Our decoding rule is closest to CAVR's logit contrast, so we state the difference precisely. CAVR contrasts every token against a stream that drops the history. We contrast only provenance-flagged tokens, and against a stream that drops the image. The question we ask is whether the image vetoes a copied word, not whether the history inflates a word. The gating is the hypothesis under test. If ungated contrast does as well, our claim about copying is wrong or unhelpful, and the ablation below is designed to show that.

Two decoding papers supply the comparison methods and constrain the design. Self-Introspective Decoding builds a contrastive branch from the least important visual tokens. SHARP steers middle-layer hidden states along directions separating faithful from hallucinated states. SHARP derives those directions from clean visual QA, and whether they survive heavy prompt noise is open. Both report that contrast with random image noise (VCD) is blunt and roughly doubles cost, which rules out a naive design that perturbs everything.

On the evaluation side, CLAIR replaces n-gram caption metrics with an LLM judge comparing candidates to reference captions. CaptionQA scores captions by whether a text-only reader can answer questions from them. Both evaluate clean, single-image settings. POPE converts hallucination into yes/no probes with adversarial co-occurring negatives and gives us the probe format. Lexical Repetitions Lead to Rote Learning shows that overlap between training and test summaries inflates ROUGE while producing "remembered hallucinations." That is the closest precedent for treating overlap as a trace of where an error came from, and we move it from training-set provenance to prompt provenance.

Research Program

The noise grid. We construct the stress test on POPE's COCO, A-OKVQA and GQA image–object pairs. For each probe about an absent object, we prepend a passage formatted like a retrieved document and vary two factors independently. The first factor is lexical: the passage names the absent object, names it under negation, names only a synonym, or does not name it. The second factor is semantic: the passage is about the same kind of scene (a kitchen passage for a kitchen photo) or about something unrelated. Passages are generated with a text model under templates and then length-matched. Present-object probes receive the same treatment, so we can check that the model is not simply becoming more skeptical. The synonym cell is diagnostic. If semantic relatedness drove errors, synonyms would hurt nearly as much as exact names. If copying drives errors, they would not.

Open-ended and realistic arms. For free-form generation, we add the same passages to COCO captioning prompts and score with CHAIR, CLAIR, and CaptionQA's reader protocol. For dialogue noise, we reuse MM-Snowball trajectories and ask what fraction of turn-4 and turn-5 errors share n-grams with the turn-3 trigger. For retrieval with useful context, we use Dyn-VQA, where the retrieved text is sometimes needed. That arm checks that the correction does not suppress legitimately copied facts. The $[\cdot]_+$ term exists so that copying is allowed when the image is silent or agrees.

Provenance analysis. For every hallucinated object or attribute in open-ended outputs, we label it as context-traceable (exact or near n-gram match in the prompt) or prior-driven (no match). We then report how the share of context-traceable hallucinations moves across the grid and across model families: LLaVA-1.5 7B, LLaVA-Next, and a Qwen vision-language model. We also test Gap 5 from the file directly. We fit SHARP-style linear probes at layers 10–15 on clean data and measure their separation (AUROC) on our noisy prompts.

Metric blindness. We pick a subset where distractor passages share vocabulary with COCO reference captions. On that subset we compare CIDEr/ROUGE against CHAIR and CLAIR as the noise level grows. The prediction is that n-gram metrics stay flat or rise while CHAIR worsens. That would mean overlap-based leaderboards cannot see this failure mode.

Baselines

The naive reference point is greedy decoding of each model with the noisy passage present and no intervention. We also run the same model with the passage removed. The no-passage condition is the ceiling a perfect noise filter could reach on POPE-style probes. The gap between the two conditions is the damage to be repaired. No reported result was found on Lacuna for POPE under prepended textual distractors, so both numbers are measured here.

The strongest existing decoding method is Self-Introspective Decoding, which reports LLaVA-1.5 7B rising from 79.1% to 83.3% POPE adversarial accuracy on clean prompts (SID). SHARP is a close second, reporting LLaVA-Next adversarial POPE accuracy of 82.8% versus 79.5% and LLaVA-v1.5 CHAIRs falling from 52.8 to 34.8 (SHARP). We run both with their released hyperparameters and give each the same budget for tuning $\alpha$-like strength settings on a held-out split of the noise grid as we give ourselves. Beating them means higher adversarial POPE accuracy and lower CHAIRs on the noisy grid at equal or lower wall-clock time, while staying within one point of their clean-prompt numbers.

For dialogue noise, the strongest competitor is CAVR from MM-Snowball. It reports a flattened decay curve but no raw numbers in the file, so we reimplement its logit-level component (history-free contrast) under the same compute. For detection, we compare against HalTrapper's Internal Grounding detector. It reports 5–12% AUROC over PoScore, with no absolute value in the file. We also compare against a SHARP-style probe, all on identical labeled hallucination sets.

The key ablation removes the provenance gate ($g \equiv 1$) while keeping the image-free contrast. It isolates whether "copied from context" is the right place to intervene. A second ablation keeps the gate but contrasts against a context-free branch, as CAVR does, which isolates the choice of what to drop from the comparison branch.

Why not just…?

Why not just filter or compress the retrieved text before the model sees it? Ext2Gen reports that standalone compressors such as Recomp, CompAct and EXIT drop needed evidence, and that training the generator to extract jointly does better, moving from 0.473 to 0.557 average score on Llama3.1-8b-instruct (Ext2Gen). There is a sharper problem specific to our hypothesis. Relevance filters keep passages that share words with the query, which are exactly the passages we predict are most harmful. We include a query-overlap filter and an LLM compressor as baselines on the grid. If either closes the gap to the no-passage ceiling on POPE while preserving Dyn-VQA accuracy, decoding-time intervention is unnecessary, and we would report that.

Why not just apply contrastive decoding everywhere, as VCD or CAVR do? This is the real threat. SID and SHARP already show that blanket perturbation costs about twice the compute (SHARP reports 7.04s versus 13.97s per sample for VCD). The open question is whether ungated contrast is less accurate, not merely slower. The gate-removal ablation settles it. If ungated image-free contrast matches the gated version on noisy POPE and Dyn-VQA, the copying hypothesis adds explanation but not method, and the contribution narrows to the diagnostic benchmark.

Why not just detect hallucinations with an internal probe like SHARP's or with HalTrapper? Probes may well work, but SHARP's directions come from clean HaloQuest data, and nobody has checked them under heavy prompt noise. HalTrapper targets self-generated context. We measure both on the same noisy labels as the provenance flag. If a probe wins on AUROC, we test whether adding the provenance flag as a feature still improves it. A cheap string feature that adds information beyond hidden states would itself be worth knowing.

Why not just evaluate with CLAIR or CaptionQA and skip the n-gram analysis? We use both as primary evaluators. The n-gram analysis is not offered as a metric. It answers a question the field still faces in practice: whether reported CIDEr and ROUGE numbers for retrieval-augmented captioning hide this failure. CLAIR itself shows BLEU@1 reaching only 0.399 Kendall's $\tau$ against human rankings on MS-COCO, where CLAIR reaches 1.000 (CLAIR). We expect that gap to widen under vocabulary-sharing noise.

Proposed Experiments and Decisive Checks

The first decisive check is the grid result. The copying claim is supported if exact-name distractors raise false-"yes" rates on absent objects substantially more than same-topic distractors that omit the name, and more than synonym distractors. It is refuted if same-topic, name-free passages hurt as much as name-bearing ones, because that would mean semantic priming rather than copying is the driver. The negation cell gets a separate verdict: a higher false-"yes" rate under "there is no X" than under no mention would show that the model, like ROUGE, ignores negation and responds to the keyword.

The second check is provenance share. In open-ended captions under noise, we expect most new hallucinations (those absent in the no-passage condition) to be context-traceable. In MM-Snowball, we expect turn-4 and turn-5 errors to overlap the turn-3 trigger far more than chance vocabulary overlap would predict. A low share would mean the noise mainly weakens visual attention in general, as visual fading suggests, rather than supplying the content of errors.

The third check is the method comparison on noisy POPE, noisy COCO captioning, Dyn-VQA and MM-Snowball. It is reported with adversarial POPE accuracy and F1, CHAIRs and CHAIRi, CLAIR, CaptionQA reader accuracy, Dyn-VQA F1-Recall (OmniSearch with GPT-4V reports 50.03%, which serves as the system-level reference rather than a like-for-like competitor), and per-sample latency. The method succeeds if it recovers most of the gap to the no-passage ceiling, beats SID and SHARP on noisy inputs at lower cost, does not reduce Dyn-VQA accuracy where retrieved text is needed, and beats the ungated ablation.

The fourth check is metric blindness. On the vocabulary-sharing subset, CIDEr and ROUGE should fail to fall, and may rise, as CHAIR worsens. Their rank correlation with CLAIR across noise levels should collapse. That would turn a long-standing complaint about overlap metrics into a specific, measurable risk for retrieval-augmented vision-language systems.