Where Hallucination Lives in the Weights: Contrastive-Saliency Sparse Tuning for Vision-Language Models

Vision-language models describe images fluently, but they often mention objects that are not there or get colors, counts and relations wrong. Most recent fixes that look inside the model do not change it. They nudge its internal activations at run time, or they rewrite its attention maps on every forward pass. The fixes that do train the model update all of its weights, which risks erasing unrelated skills. This proposal tests a middle path. We would find the small set of existing weights, well under one percent, whose gradients respond to "hallucinated versus correct phrase" and not to ordinary captioning. We would train only those weights, freeze everything else, and then examine where in the network the selected weights sit.

The payoff is practical and scientific. On the practical side, the fix becomes a permanent weight edit, with no added modules and no run-time machinery, and it should forget less than dense tuning. On the scientific side, the location of the selected weights gives a direct, causal answer to an open disagreement. Different papers claim hallucination is decided in the first two layers, in the middle layers, or in the deep layers.

Core Claim

We claim that anti-hallucination tuning in vision-language models has a small, identifiable weight footprint. Selecting parameters by how strongly their gradient separates hallucinated from grounded phrases, discounted by how strongly they matter for general captioning, gives three results at once. It reduces hallucination about as much as dense contrastive tuning. It preserves general multimodal and text ability better than dense tuning, LoRA with the same number of trainable parameters, or a random sparse mask of the same size. It also reveals where the correction must happen, including in the vision-to-language projector, which activation steering cannot reach.

The closest prior work falls into two groups. The first is sparse tuning in non-multimodal settings. Gradient-based Parameter Selection (GPS) picks top-K input connections per neuron for vision classifiers. Supervised Pinpoint Tuning tunes the roughly 4% of attention heads it causally links to sycophancy in text-only models. The second is dense hallucination tuning of vision-language models, such as HALVA. Neither group has been combined, and neither selects weights with a hallucination-specific contrastive gradient signal.

Research Program

The selection signal is the key ingredient. GPS found that plain cross-entropy gradients are noisy for choosing weights, and it switched to a contrastive loss. We make the analogous move for generation. We reuse HALVA's phrase-level augmented pairs, where a caption is paired with a copy in which one grounded phrase is swapped for a hallucinated one. From these pairs we compute the per-weight gradient of the log-probability margin between the grounded and the hallucinated phrase, accumulated over a few hundred pairs.

Separately, we accumulate gradients of ordinary caption likelihood on unaltered instruction data. The motivation comes from GRAIT, which found near-orthogonal gradient directions for distinct behaviors in text LLMs, with an inner product of about 0.008. If a similar separation holds here, hallucination-relevant weights can be chosen apart from generally useful ones. The score for weight $i$ is

$$s_i = \frac{|g_i^{\text{hall}}|}{\bar g^{\text{hall}}} - \lambda \frac{|g_i^{\text{gen}}|}{\bar g^{\text{gen}}},$$

where $g_i^{\text{hall}}$ and $g_i^{\text{gen}}$ are the accumulated gradients from the two objectives, the bars denote their means over all weights (for scale matching), and $\lambda \ge 0$ trades hallucination relevance against general relevance.

Following GPS, we keep the top-K connections per neuron. The candidate pool deliberately spans the vision encoder's last blocks, the projector, and every attention projection and MLP in the language model. This lets the selection itself decide where the correction lives. We then train only the masked weights with HALVA's phrase-level loss and its token-level KL term.

Two analyses follow training. The first maps the mask by layer and module to see whether it concentrates in shallow layers, middle layers, deep layers, or the projector. Each position corresponds to one account in the literature:

  • EVAS argues for the sink heads of layers 1–2.
  • SHARP argues for layers 10–15, where faithful and hallucinated states become linearly separable.
  • REVIS argues for deep layers.

The second analysis retrains with masks restricted to each region at the same budget. This turns the three descriptive accounts into a single causal comparison.

Baselines

The naive reference is the untuned model under greedy decoding, together with dense HALVA-style tuning. The latter shows what the data can do when budget is unrestricted.

The strongest existing training method is HALVA's data-augmented phrase-level alignment, run densely on the same pairs with its KL regularizer. It is the strongest competitor because it already targets both hallucination and the degeneration that coarse sequence-level alignment causes. The file reports up to a 13.4% AMBER F1 improvement and a 4.2% reduction in description hallucination rate. We will rerun it rather than copy numbers, using the same base model, data, steps and learning-rate grid.

The strongest inference-time references on LLaVA-1.5 are two steering methods:

  • EVAS, reported to reduce CHAIR sentence score from 47.0 to 36.4.
  • SHARP, reported to reduce it from 52.8 to 34.8.

Their base numbers differ, which suggests different decoding settings. We will therefore rerun both under one protocol.

LoRA is the obvious parameter-efficient competitor. We will set its rank so its trainable-parameter count equals the mask size, and give it the same data, steps and tuning grid. The file contains no reported LoRA result for LVLM hallucination.

Three ablations isolate the proposal's ingredients:

  • A random mask of equal size and per-neuron structure tests whether selection matters or sparsity alone regularizes.
  • Setting $\lambda=0$ (hallucination saliency only) tests the general-gradient discount.
  • A plain caption cross-entropy saliency tests the contrastive selection signal.

Beating the field means two things. First, the method must reach hallucination scores within about one point of dense HALVA on CHAIR and AMBER. Second, it must show a smaller drop than dense HALVA, LoRA and the random mask on MME, MMBench and text reasoning. The margin must exceed seed variance across three seeds.

Why not just…?

Why not just use LoRA? This is the most serious threat. GPS's advantage over LoRA was shown on unimodal classification (91.78% on FGVC at 0.77% of parameters), not on generative hallucination. That evidence does not transfer automatically. The equal-count LoRA baseline settles the question directly. If LoRA matches on both hallucination and retention, we drop the practical claim and keep only the localization findings.

Why not just use activation steering, or fold the steering vector into a bias? A constant steering vector added at one layer is exactly equivalent to a bias in that layer's output projection, so folding it into the weights is free. We include this folded-bias baseline. Its limits come from the methods themselves. SHARP and REVIS gate their vectors by input-dependent risk scores, and EVAS rewrites attention maps in an input-dependent way; a fixed bias captures neither. REVIS also reports collapse at high steering intensity, where the model outputs the literal string "image.jpg". Finally, none of these methods can repair information lost before the language model. We test that last point below.

Why not just tune densely with a KL penalty, as HALVA does? HALVA's KL term controls fluency but does not guarantee retention of unrelated skills. In text LLMs, full SFT dropped GSM8K accuracy from 24.7% to 14.6%, while pinpoint tuning kept it at 23.5% (Supervised Pinpoint Tuning). Whether a similar gap appears for vision-language models under a KL penalty is exactly what the retention comparison measures.

Why not just use contrastive decoding? Visual contrastive decoding needs no training, but it roughly doubles per-token cost: 0.053s versus 0.025s, as reported in REVIS. A weight edit adds no cost at inference. We report latency alongside quality.

Proposed Experiments

The main runs use LLaVA-1.5-7B, which is shared with EVAS and SHARP, and Qwen2.5-VL, which is shared with REVIS, at mask budgets of 0.1%, 0.25% and 0.5%.

Hallucination is measured on four benchmarks:

  • POPE, across random, popular and adversarial splits on COCO, A-OKVQA and GQA. We also track the "yes" rate, since the POPE paper documents rates near 99%.
  • CHAIR, at sentence and instance level, with object recall reported so that shorter, emptier captions cannot pass as improvements.
  • AMBER.
  • FGHE, for attributes and relations.

Retention is measured on MME, MMBench, and GSM8K plus MMLU to capture the language model's text ability.

We will run four decisive checks.

The first check is the retention–hallucination frontier. The contrastive-saliency mask should dominate dense HALVA, equal-count LoRA and the random mask. If the random mask matches it, the conclusion is that sparsity regularizes, not that selection finds anything. We would report that outcome as such.

The second check is localization. The mask map and the region-restricted retraining will show which of layers 1–2, 10–15, deep layers or the projector carries most of the achievable reduction. A result where no single region suffices while the unrestricted mask does would itself be informative. It would suggest that the three accounts describe parts of one circuit rather than competing locations.

The third check targets perceptual blindness. We use a linear probe on visual tokens at the projector output to find POPE items where the queried object's presence is not decodable. On those items, we compare accuracy gains from SHARP, REVIS and our method. The prediction is that steering gains nothing there and that masks including projector weights do. If our method also gains nothing, the "reaches what steering cannot" claim fails.

The fourth check is gradient separation. We measure the cosine similarity between $g^{\text{hall}}$ and $g^{\text{gen}}$ per module, and sweep $\lambda$. Near-orthogonality combined with a retention benefit from $\lambda>0$ would support the GRAIT-motivated discount. High overlap would mean the discount is ineffective and should be removed.