When a vision-language model answers a question about an image with help from retrieved documents, a wrong answer can come from two very different places. Sometimes the retriever, the fast first-stage search that narrows millions of documents to a few dozen, never returned the right document, so the model had nothing to work with and fell back on guesses. Other times the right document was in front of the model, and the model's own memorized associations overwrote it. These two failures call for opposite fixes. The first needs more or different retrieval. The second needs the model to rely more on what it was given. Today's pipelines cannot tell which one happened, so they either always pay for extra retrieval or always apply a blanket "trust the context" correction.
This note proposes to separate the two failures causally and then cheaply. The method is activation patching: copying a model's internal activations from one run into another and checking whether the output changes. We use it in a setting where the "corrupted" run comes for free, namely the same question answered with and without the correct document in the candidate pool. That analysis tells us which attention heads and feed-forward layers carry the retrieved evidence and where it gets overwritten. We then distill it into a single-pass internal signal that routes each query to the right repair: expand retrieval when the evidence was missing, or amplify the context pathway when the evidence was present but overridden. The claim is that this router beats both fixed strategies on knowledge-based visual question answering at matched compute, and that no black-box confidence signal can do the same routing.
Core Claim
In knowledge-based visual question answering (KB-VQA, where answering requires an external fact about the pictured entity, such as a building's construction year), hallucinated answers split into a retrieval-miss class and a parametric-override class. These classes leave distinguishable signatures in the generator's internal activations: how much attention the answer position pays to retrieved-text tokens versus image tokens, and how much the late feed-forward layers push the output away from what the attention pathway carries. Because the classes need opposite repairs, a router built on these signatures should beat any single repair. It should also beat routers that use only output confidence or reranker score, since those signals indicate that an answer is risky but not why.
Where This Sits
Three lines of work supply the pieces, and none connects them. On the mechanism side, ReDeEP shows in text-only retrieval-augmented generation that hallucinations arise when "knowledge FFNs" overwrite what "copying heads" move from context. In roughly 65–77% of hallucinations, the heads attended to the right context. ARC-JSD traces answers to retrieved sentences by ablating sentences and measuring the resulting change in the output distribution. Both assume the right document was already retrieved, which is exactly the assumption we drop. Both are also text-only.
On the multimodal side, DMAS steers attention heads in LLaVA-v1.5 and Qwen-VL along "truthfulness" and "visual perception" directions. Why LVLMs Are More Prone to Hallucinations in Longer Responses finds that attention drifts away from image tokens over a long generation. These papers motivate the proposal by exposing an unresolved conflict: does multimodal hallucination come from attention detaching from evidence, or from feed-forward layers injecting memory over evidence that was attended to? The two papers disagree on this point. Neither considers a retrieval stage.
On the retrieval side, EntRAG improves the candidate stage for KB-VQA with entity-level image–text binding. Reranker-Guided Search recovers documents ranked thousands of places down by walking a document-similarity graph under reranker guidance. Both treat the generator as a black box. They therefore cannot tell whether a final error was a retrieval failure at all.
Our proposal takes its causal-tracing method from Meng et al. and Wang et al.. It takes its statistical standards for localization claims from the Predict–Intervene–Validate pipeline. The application regime comes from EntRAG's benchmarks, and the two repairs come from Reranker-Guided Search and ReDeEP's attention-up, FFN-down intervention (AARF).
Research Program
The first stage is a causal diagnosis that exploits the candidate bottleneck as a natural intervention. For each E-VQA and InfoSeek question we build two candidate pools of equal size: the retriever's actual top-$k$, and the same pool with the gold entity article swapped in for the lowest-ranked item (or kept, if already present). Running the generator (LLaVA-v1.5 and Qwen-VL, matching DMAS) on both pools yields paired clean and corrupted runs that differ only in whether the evidence exists.
Errors where the gold article was absent and swapping it in fixes the answer are labelled retrieval-miss. Errors where the gold article was already present and the answer is still wrong are candidates for override. For these, we patch activations from a "gold-only context" run into the full run, one component at a time, across image-token heads, retrieved-text heads, and feed-forward layers at the answer position. We measure restoration as
$$R(c) = \frac{p_{\text{patched}(c)}(y^*) - p_{\text{corrupt}}(y^*)}{p_{\text{clean}}(y^*) - p_{\text{corrupt}}(y^*)},$$
where $c$ is the patched component, $y^*$ is the gold answer token, and $p$ is the model's probability of it under each run. A component with $R(c)$ near 1 carries the evidence on its own. Path patching between the top heads and downstream feed-forward layers then tests whether override happens through attention detachment or through feed-forward overwriting. This directly adjudicates the conflict between ReDeEP and the long-response findings in the multimodal retrieval setting. Localization claims will be reported with Predict–Intervene–Validate-style held-out validation rather than single-split rankings.
The second stage distills the diagnosis into a cheap signal. Patching costs many forward passes per query, so it cannot run at inference. From the components identified in stage one we compute single-pass features. The first is the attention mass from the answer position onto retrieved-text tokens and onto image tokens, restricted to the localized heads. The second is a ReDeEP-style divergence between the logit-lens readout before and after the localized feed-forward layers. A small classifier trained on stage-one labels predicts miss, override, or correct.
The third stage is the router. Queries flagged as misses trigger Reranker-Guided-Search expansion over a proximity graph of the entity knowledge base. Queries flagged as overrides get AARF-style amplification of the localized heads and damping of the localized feed-forward layers, with the intervention strength fixed on a validation split, since DMAS reports steep degradation when strength is too high. Queries predicted correct are answered as is.
Baselines
The naive reference point is the single-pass pipeline with no repair: the retriever's top-$k$ fed to the generator. We also run the strongest existing pipeline for this task, EntRAG, which reports VQA accuracy of 46.1 on E-VQA and 44.5 on InfoSeek (EntRAG). Where we cannot reproduce EntRAG's full stack, we use its reported numbers only as a reference and compare our router on top of our own reimplemented retriever. The router counts as beating the field if it adds accuracy over its own no-repair pipeline beyond seed variance and closes a meaningful share of the gap to the gold-swap oracle.
The two fixed-repair baselines are the real competitors, because each solves one failure class. The first applies Reranker-Guided-Search expansion to every query. On M-BEIR that method improves the reported score from 25.9 to 31.0 (Reranker-Guided Search), but no reported result was found on Lacuna for it on E-VQA or InfoSeek. The second applies the AARF intervention to every query. For fairness, all three systems share the same generator, retriever, graph, and knowledge base. They also share the same validation set for tuning intervention strength and expansion budget, and they are compared at matched total reranker and generator forward passes. The router must beat both fixed repairs at equal compute, or match them at clearly lower compute.
Two ablations isolate the key ingredient. The first replaces our mechanistic features with black-box ones (answer entropy, top reranker score, and their combination) feeding an identical classifier. The second keeps the mechanistic feature form but computes it on randomly chosen heads and layers of matched count, which tests whether patching-based localization matters. For detection quality alone, the closest reported number is ReDeEP's chunk-level AUC of 0.7458 on RAGTruth with LLaMA2-7B, against 0.7290 for RAGAS (ReDeEP). That figure is text-only, and no reported result was found on Lacuna for mechanistic hallucination detection on KB-VQA.
Why not just…?
Why not just retrieve more documents, or run graph search on every query? Expansion helps only the miss class. Per ReDeEP, most text hallucinations happen with the right context already present, and extra candidates add distractors and latency without changing the generator's tendency to override. The always-expand baseline at matched compute tests this directly. If the override class turns out to be small in KB-VQA, always-expand will be hard to beat, and the stage-one class proportions will reveal that before the router is ever built.
Why not just gate repairs on output confidence or reranker score? This is the most serious threat. Low confidence plausibly flags risky answers well. Our bet is that it cannot tell which repair is needed: an overridden answer may be confidently wrong, and a missing-evidence answer may be confidently guessed from memory. The black-box-router ablation settles this. If it matches the mechanistic router on three-way classification and end-to-end accuracy, the mechanistic component is unnecessary and we will report that.
Why not just fine-tune or instruction-tune the system to use context better? Instruction tuning fixes format errors but not coverage. UniIR cuts wrong-modality retrieval from about 59% to about 3% yet still reaches only 48.9% Recall@5 on M-BEIR (UniIR), and EntRAG shows that generic CLIP features keep missing fine-grained entities. On the generator side, alignment training tends to buy conservatism rather than accuracy. Tuning also does nothing for queries whose evidence was never retrieved, so it cannot replace the miss branch of the router.
Why not just use contrastive decoding against a corrupted image? Contrastive decoding targets over-reliance on language priors relative to the image. It does not address retrieved text being overwritten, and DMAS reports it to be about twice as slow as activation steering. We include it as a secondary reference on the override class only.
Proposed Experiments
The first decisive check is whether the diagnosis is real. On E-VQA and InfoSeek errors we report the share of misses versus overrides. Within the override class, we report whether restoration $R(c)$ concentrates in a few retrieved-text heads (attention detachment) or in late feed-forward layers (overwriting). We compare this split against ReDeEP's 65–77% figure. As a guard against hidden hallucinations (Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models), we also check the "correct" class by testing whether ablating the retrieved context or the image leaves correct answers unchanged.
The second check is the three-way classification itself. It reports macro-F1 and per-class AUC for the mechanistic router against the black-box and random-component ablations on held-out questions and held-out entities. It also tests transfer from LLaVA-v1.5 to Qwen-VL with re-localization, since circuits are known to vary across architectures.
The third check is end-to-end. We report VQA accuracy on E-VQA and InfoSeek for no-repair, always-expand, always-AARF, black-box routing, and mechanistic routing, all at matched forward-pass budgets, alongside the gold-swap oracle as a ceiling. As a side-effect check, we run POPE and MME to confirm that the override intervention does not degrade general perception, given DMAS's warnings about steering strength.
The proposal fails cleanly in either of two cases: misses and overrides turn out not to be separable from internal activations beyond what output confidence already captures, or one class dominates so heavily that a fixed repair is enough.
References
- ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability
- Attributing Response to Context: A Jensen–Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
- Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Beyond Sequential Reranking: Reranker-Guided Search Improves Reasoning Intensive Retrieval
- Locating and Editing Factual Associations in GPT
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
- A Statistical Framework for Mechanistic Claims in Neural Networks: The Predict--Intervene--Validate Pipeline
- UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models