Vision-language models (systems that feed image features into a large language model and generate text about the image) often report what is linguistically likely instead of what is in the picture. Shown a blue bear, they say the bear is brown. Recent mechanistic work traces this "linguistic prior" failure to a specific depth. In LLaVA-1.5, the correct visual answer forms around layer 20 and is then overwritten in the last layers, where frequent text associations win (Towards Understanding How Knowledge Evolves in Large Vision-Language Models). Skipping those late layers removes the error. Multilingual interpretability work points at the same final layers for a different reason: they decide which language the model writes in. Tuning only the last 3–5% of parameters controls output-language consistency (LinguaMap). So the fix for prior override seems to remove exactly the machinery a Hindi or Russian answer depends on. Every study of image-versus-memory conflict to date has been English-only, so nobody has noticed the collision.
We propose to measure this collision and to resolve it. The working hypothesis is that, inside the same late layers, the update that overwrites visual evidence with the text prior and the update that selects the output language can be pulled apart, both in depth and in a low-dimensional subspace. If so, a model can be stopped from "un-seeing" the image without losing the ability to answer in the user's language. The proposal treats parametric-versus-contextual knowledge conflict as the core phenomenon, with the image playing the role of the context. Linguistic prior dominance is the failure mode under study, hierarchical language specialization is the constraint the fix must respect, and multilingual hallucination benchmarks are the evaluation setting.
Core Claim
We make two claims, one diagnostic and one corrective.
The diagnostic claim is about non-English queries. When the image contradicts common-sense knowledge and the question is not in English, the model loses the conflict more often, and it loses it in a particular way. The visual answer is still decodable in the middle "semantic hub" layers, where concepts from all languages and from image patches converge onto English-like representations (The Semantic Hub Hypothesis). It is then erased during the late band of layers, before the language-control layers. The predicted signature has two parts. First, the visual answer peaks at roughly the same depth in English and non-English. Second, the late-layer drop toward the prior answer is larger for non-English queries. This is a falsifiable alternative to the account given by CLAIM, which locates multilingual failure earlier, as attention drift away from image patches in layers 10–25. If CLAIM's account is complete, the visual answer should never form in the middle layers for non-English queries. We will test which account holds, language by language.
The corrective claim is that a small intervention can suppress the late override while leaving language selection untouched. The intervention is gated by how likely the input is to involve a conflict. We define the update a late layer $l$ writes to the residual stream (the running hidden state each layer adds to) as $u_l$. We define $P$ as the projection onto a "language subspace" estimated from parallel prompts that differ only in language. We define $g \in [0,1]$ as a conflict score read from a linear probe at a middle layer. The edited update is then
$$ \tilde u_l = P u_l + (1 - \alpha g)(I - P) u_l, \qquad l \in \mathcal{M}, $$
where $\mathcal{M}$ is the set of layers between the stabilization layer and the final language-control layers, and $\alpha$ is a strength set on a development split. The language component always passes through unchanged. The remaining component, which is where we hypothesize the prior override lives, is damped only when the probe signals a likely conflict. The gate answers a stated limitation of fixed-strength steering (ContextFocus). It is plausible because conflict is linearly detectable well before the output: probes reach about 90% accuracy at layer 14 in text LLMs (Analysing the Residual Stream of Language Models Under Knowledge Conflicts).
How This Differs from Closest Work
Four papers come closest, and each leaves out one piece this proposal needs.
When seeing Overrides Knowing is the nearest neighbor. It finds that MLP blocks favor memorized facts while specific upper-layer attention heads carry visual counterfactual evidence, and it steers between them on WHOOPS-AHA!. It is English-only and uses single-token completions, so it never confronts language control. We reuse its head-finding procedure to ask whether the same counterfactual heads fire for non-English queries.
CLAIM is multilingual. It shifts non-English attention toward English grounding patterns in the middle layers. It targets object existence (POPE), not conflicts with world knowledge, and it does not touch the late layers. Our method intervenes in a different depth band, on a different failure, so the two are complementary and should combine.
Towards Understanding How Knowledge Evolves supplies the mutation-layer finding and the layer-skipping fix, but only in English. Its fix is a strong ablation baseline for us, and our first prediction is that it breaks language consistency.
When Abundance Conceals Weakness (the CLEAR benchmark) shows that cross-lingual knowledge conflicts behave differently across scripts, but only in text. We test whether its "script barrier" survives when the evidence is an image, which is language-free.
MMKC-Bench motivates the problem. In English, multimodal models detect conflicts 75–79% of the time yet still answer from memory, and larger models are more stubborn. That gap between detecting and acting is what the probe gate exploits.
Research Program
The first component is a multilingual conflict test set. We translate the queries and both candidate answers of WHOOPS-AHA! and Visual CounterFact into the seven CLAIM languages (Chinese, Spanish, Russian, Portuguese, Bulgarian, Hindi, German). We add Bengali, Japanese and Swahili from CLEAR to cover script and resource extremes. Native speakers verify that each translated answer pair stays unambiguous. Images stay identical across languages, so any cross-language difference is attributable to the query language alone.
The second component is the depth analysis. For each model and language we decode intermediate states with early exits, as Wang et al. do. Logit Lens projections are known to be distorted, so we repeat the analysis with tuned per-layer readouts and treat agreement between the two as the criterion. We record the probability of the visual answer and of the prior answer in both the target-language and English vocabularies. This yields a per-language curve showing where the visual answer forms, where it peaks and where it is lost. We also locate counterfactual heads in English and measure their activation and image-attention contrast under each other language.
The third component is the intervention. We estimate $P$ from parallel image captions in the eleven languages, using differences of late-layer updates across languages with content held fixed. We fit the conflict probe on English conflict and non-conflict pairs only, so that transfer to other languages is itself a test. We compare gated, ungated and probe-free variants.
Models are LLaVA-1.5-7B, where the mutation stage was mapped, and Qwen2.5-VL-7B, which is stronger multilingually and was tested on MMKC-Bench. Gemma3 is a third model if compute allows, since the counterfactual-head results exist for it.
Baselines
The naive reference point is the unmodified model with a plain instruction to answer from the image in the query language. All methods share the same images, translated prompts, decoding settings and a fixed development split for hyperparameters. Each tunable method gets the same budget of 20 settings.
The strongest existing method for multilingual hallucination is CLAIM. It reports an average 13.56% improvement on multilingual POPE (up to 30% in Spanish) and 21.75% on MME hallucination subsets across the seven languages. We rerun it with its calibration procedure on our models. No reported result was found on Lacuna for CLAIM, or any method, on multilingual WHOOPS-AHA! or multilingual Visual CounterFact. Those numbers will be produced here.
For English conflict steering, the competitor is factual/counterfactual head scaling from When seeing Overrides Knowing. That paper reports 74% factual accuracy in LLaVA-NeXT when boosting factual heads, but that is the opposite direction to ours, so we rerun it boosting counterfactual heads instead.
Visual Contrastive Decoding (Leng et al.) reports up to +7.4 F1 on POPE MSCOCO random and raises LLaVA-1.5's MME total from 565 to 604. We run it in every language. Our prediction is that its noise-induced prior is English-shaped, so subtracting it distorts non-English output.
Three ablations isolate the ingredients. The first is Wang et al.'s plain layer skipping of the whole late band, which tests whether language-control preservation matters. The second is skipping all late layers except the final two, which tests whether depth alone separates the two functions. The third is our method with $P = 0$ and with $g \equiv 1$, which tests the subspace and the gate respectively.
Beating the field means raising the visual-answer rate on non-English conflict items above CLAIM and above head steering. At the same time, language consistency must stay within 2 points of the unmodified model, and POPE and general MME scores must not fall below the unmodified model.
Why not just…?
Why not just translate the query to English, run the model, and translate back? This is the most serious threat. It might close most of the language gap. It does not address the English prior override itself, which MMKC-Bench shows is severe and grows with scale. The fair competitor is therefore translate-test combined with English layer skipping, and we run exactly that. No reported result was found on Lacuna for translate-test on multimodal conflict. If that pipeline matches our single-pass method on visual-answer rate, our corrective claim fails and the contribution reduces to the diagnosis. We would still check script-barrier languages, where CLEAR suggests translation of entity-centric content is fragile, and report two extra machine-translation passes as a cost.
Why not just skip the late layers but keep the last one or two? LinguaMap places language control in the final 1–2 layers, so this depth-only split might be enough. That would make the subspace projection unnecessary. It is ablation two above. If it matches the full method on both visual-answer rate and language consistency, we adopt it as the method and drop $P$. We report it either way.
Why not just apply CLAIM, since it already fixes multilingual grounding? CLAIM corrects middle-layer attention drift, but its target is object existence. Under our diagnostic hypothesis, the visual answer forms and is then overwritten later, which CLAIM does not touch. The depth curves decide this directly. If CLAIM alone restores the visual answer at the output, the late-override hypothesis is wrong. We also run CLAIM plus our edit to test whether the two add up.
Why not just prompt the model to trust the image? MMKC-Bench shows that models detect conflicts yet still follow memory despite explicit evidence. Are VLMs Seeing or Just Saying? shows that "let me look again" phrases are textual habits, not real re-inspection. The instruction-prompted model is our naive baseline, so this is measured rather than assumed.
Decisive Checks
The diagnostic claim stands or falls on the depth curves. It survives only if non-English visual-answer probability peaks in the middle layers at a level comparable to English, and then shows a larger late drop. It fails if the visual answer never forms, which would favor CLAIM's attention-drift account. We report this per language, separating Latin-script from non-Latin-script languages to test whether CLEAR's script barrier appears with image evidence.
The corrective claim stands or falls on a two-axis result: visual-answer rate against language consistency. Plain layer skipping should raise the first and collapse the second. Our edit should raise the first while holding the second.
Two further checks guard against overfitting and false positives. The probe trained on English must transfer to other languages with an AUROC above 0.8, since the gate is useless otherwise. On non-conflict control images from WHOOPS!, the gated method must not reduce accuracy, which confirms that it fires only when a conflict is present.
References
- Towards Understanding How Knowledge Evolves in Large Vision-Language Models
- LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
- The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
- CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models
- When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination