Where Quantization Turns Into Hallucination in Vision-Language Models

Vision-language models describe images in text. Running them in low precision means storing their numbers in 4, 2, or even 1 bit instead of 16, which is how they fit on phones and consumer GPUs. These models already have a known weakness: they sometimes "see" objects that are not there because the language model inside them guesses what usually appears together. A table suggests chairs, so the model mentions chairs. This note asks a narrow, testable question. When we compress such a model, which compressed part converts rounding noise into invented objects, and at what point in a long answer does that happen?

Our hypothesis is that quantization-induced hallucination is mostly a margin-flip effect. Rounding noise flips the model's choice wherever the full-precision model was barely confident. Those barely-confident choices pile up late in long descriptions, because the model's attention to the image fades as it generates. If the stored memory of image tokens is compressed as hard as the memory of text, that fading speeds up. The practical consequence is simple. At a fixed memory budget, spending more bits on the cached image tokens and fewer on cached text tokens should cut long-form hallucination more than spending the same bits evenly. A single forward pass is enough, so the speed benefit of quantization is kept.

Core Claim

We make two claims, and each can be falsified. The first is diagnostic. In generative vision-language models, the chance that quantization changes a full-precision token rises sharply as that token's full-precision logit margin shrinks. The flips favor objects that co-occur with what is actually in the image, so the model falls back on language priors rather than producing random errors. Compressing the KV cache (the stored keys and values of past tokens that the model attends back to) moves the onset of hallucination earlier in the answer than compressing weights to the same total size.

The second claim is constructive. Suppose the KV-cache bits are split by modality, with image tokens kept at higher precision and generated text tokens pushed lower, and the average bit budget stays fixed. Then this split should recover most of the long-form hallucination lost to uniform cache quantization, and it should do so in one pass.

We define the margin of the token the full-precision model chose at step $t$ as $m_t = z_t(y_t^*) - \max_{y \neq y_t^*} z_t(y)$. Here $z_t$ is the full-precision logit vector and $y_t^*$ is the chosen token. The budget constraint is

$$\frac{N_v b_v + N_t b_t}{N_v + N_t} = B,$$

where $N_v$ and $N_t$ are the numbers of cached image and text tokens, $b_v$ and $b_t$ are their bit widths, and $B$ is the average bits per cached element. The constraint has teeth because image tokens often outnumber text tokens. LLaVA-style models use hundreds of image tokens, so protecting them is expensive and has to pay for itself.

What Prior Work Establishes and Leaves Open

Three lines of evidence motivate the margin view, and each comes from a different setting.

Flat Score, Amplified Failures shows that 4-bit weight quantization of text-only tool-using agents leaves headline scores flat while failures multiply. Tool-name hallucination doubled from 19.5% to 38.3% for Gemma-4-31B. The authors explain this with a logit-margin account, but they support it only behaviorally, without inspecting logits. We test that account directly, with internal margins, in a multimodal setting.

Less Precise Can Be More Reliable finds that quantization acts like a low-pass filter on weight and activation spectra. In contrastive models such as CLIP and SigLIP, this pushes them toward background shortcuts. That study is confined to classifiers, and the authors explicitly leave generative models open. We ask whether the same shift appears in generation as reliance on language priors.

Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that attention drifts from image tokens toward previously generated text as an answer grows. This is the mechanism we expect cache compression to accelerate.

Two quantization papers are the closest existing work, and both constrain the design. ImpQuant weights image tokens by importance during weight-only post-training quantization (rounding a trained model without retraining), tested at 3 and 4 bits. It reports that this reduces POPE hallucination toward the FP16 level. It never touches the KV cache or long-form generation, and the summary in our file gives no POPE numbers. CalibQuant quantizes the multimodal KV cache to 1 bit. It reports up to 10× throughput on InternVL and finds that image tokens have different channel statistics from text tokens. It handles those statistics through per-channel scaling, but it gives image and text tokens the same bit width. It also evaluates captioning and DocVQA accuracy only, with no hallucination metric and no analysis of attention drift.

Our proposal differs from both. We allocate bits between modalities in the cache under a fixed budget, and we score the result by where in the answer hallucination begins.

Training-free anti-hallucination decoding is the main competitor. Visual Contrastive Decoding and RITUAL contrast or combine predictions from clean and perturbed images, which roughly doubles per-token cost. Self-Augmented Visual Contrastive Decoding refines the choice of perturbation. SHARP steers mid-layer activations in a single pass and reports 7.04 s per sample against 13.97 s for VCD. We treat these methods as baselines run on top of quantized models, not as parts of our method.

Research Program

The first phase localizes the damage. On LLaVA-1.5, Qwen2.5-VL-7B and an InternVL model, we generate full-precision long captions on COCO images. For each token we record its margin $m_t$, its position $t$, and the share of attention it places on image tokens. We then compress one part of the model at a time while holding the others at 16 bit. The parts are the vision encoder, the projector that maps image features into the language model, the language-model weights at 4 and 3 bit, the KV cache at 4, 2 and 1 bit, and everything together.

For every configuration we rerun the full-precision prefix under teacher forcing, meaning we feed the model the full-precision tokens so far rather than its own outputs. This gives us the probability that each token flips, as a function of margin and position. We also label each flip toward an absent object by how strongly that object co-occurs with the objects that are present. If the margin account holds, flip probability should fall steeply with margin. Flips should land on co-occurring objects more often than chance. Cache compression should produce flips later in the answer than weight compression of equal byte size, matched to the same point where image attention has already decayed.

The second phase builds the allocation rule. We sweep $(b_v, b_t)$ along the budget constraint for $B \in \{1, 2, 4\}$, reusing CalibQuant's per-channel scaling and logit calibration so that only the allocation changes. We test two refinements. One gives higher precision only to the image tokens ImpQuant would mark as important. The other keeps the most recent text tokens at higher precision. These show whether the gain comes from modality itself or from token importance.

Baselines

The naive reference is the same model with uniform KV quantization at budget $B$, using CalibQuant's scheme. The gap between this and FP16 is the damage to be recovered. FP16 itself serves as the upper bound.

The strongest existing quantization method on this question is ImpQuant for weights paired with CalibQuant for the cache. This is the best published combination aimed at quantized vision-language models, and ImpQuant is the only quantization paper in our file that reports hallucination effects. Our cache allocation sits on top of ImpQuant weights, so the comparison isolates the cache change. All arms share the same 256 MSCOCO calibration images ImpQuant used.

The strongest hallucination-mitigation competitors are RITUAL and VCD run on the quantized model. RITUAL reports POPE accuracy rising from 84.13% to 88.87% on full-precision LLaVA-1.5. SHARP reports CHAIR_s on LLaVA-v1.5 falling from 52.8 to 34.8 and POPE-Adversarial accuracy on LLaVA-Next rising from 79.5% to 82.8%. We have no reported result on Lacuna for any of these methods on quantized models, so we will measure them. We compare under two regimes, equal memory and equal wall-clock time per sample on the same GPU. The second regime is generous to us, because the contrastive methods pay a second pass. We also test SHARP steering vectors, extracted at full precision and applied to the quantized model, as a single-pass competitor.

A token-dropping baseline keeps half of the image tokens at 16 bit. We choose the half by ImpQuant's redundancy score and match the bytes to our allocation.

Beating the strongest baseline means one of two results. At equal memory and single-pass latency, we must recover at least half of the CHAIR and POPE-Adversarial gap between uniform KV quantization and FP16 without losing COCO Caption or DocVQA accuracy. At equal wall-clock time, we must match or beat RITUAL on the quantized model.

Why not just…?

Why not just quantize less, keeping the whole cache at 4 or 8 bit? That is the right answer if memory allows. The budget-matched framing exists because at long contexts the cache dominates memory, as CalibQuant's motivation shows. Our claim is about how to spend a fixed number of bits. The $B=4$ arm of the sweep shows directly whether the effect disappears once bits are plentiful, and we will report it if it does.

Why not just run RITUAL or VCD on the compressed model? They may well reduce hallucination more. But their second pass roughly doubles per-token cost, and SHARP measured VCD at about twice SHARP's time per sample, which cancels the 1.45× speedup ImpQuant reports. The equal-wall-clock comparison settles this. If RITUAL wins even at matched time, the honest conclusion is that cache allocation is a complement to contrastive decoding, not a substitute, and we will test the two stacked.

Why not just use ImpQuant, since it already protects important image tokens? ImpQuant shapes weight rounding using image-token importance. It leaves cache precision uniform and does not test long-form generation. If phase one shows that weight error alone explains the position-dependent flips, our allocation adds nothing, and the cache-only versus weight-only localization will reveal this before any method is built.

Why not just drop redundant image tokens instead of compressing them? This is a real threat. Token dropping saves memory without adding rounding noise to the tokens that remain. The byte-matched dropping baseline tests it. If dropping matches our allocation, we will reframe the contribution around that finding.

Why not just fix it with prompting or self-critique? MERLIM reports swings of up to 26.5% from rephrasing alone, and work on vision-language models as evaluators (Chen et al.) shows that self-critics invent errors of their own. We include one fixed "describe only what is visible" prompt as a cheap control but do not expect it to separate the arms.

Proposed Experiments and Decisive Checks

Hallucination is measured with POPE on its random, popular and adversarial splits. The adversarial-minus-random gap serves as our measure of language-prior reliance. We also use MME existence and count. For long captions we use CHAIR_i and CHAIR_s, broken out by token-position bins, along with AMBER from the HalTrapper suite. Hidden hallucination is measured with MERLIM's inpainted images, where an object has been erased and a model that still names it is relying on scene context. That benchmark reports an average precision drop of about 8% even at full precision, and we ask whether quantization widens it. Utility is measured with COCO Caption, DocVQA and TextVQA. Efficiency is measured as tokens per second and peak memory with real Triton kernels at 1, 2 and 4 bit. Following the warning in the file about simulated quantization, arms that are only simulated are labeled as such.

Three results would be decisive. The first tests mechanism. If flip probability does not depend on margin, or if flips are not biased toward co-occurring objects, the margin account fails, and with it the reason to expect position-dependent damage. The second tests the cache claim. If uniform and modality-split caches at the same $B$ produce indistinguishable CHAIR-by-position curves, the allocation idea fails. The third tests competition. If byte-matched token dropping or equal-time RITUAL dominates the split cache, we report that and redirect the work.

A positive outcome would give the first measured account of how compressing each part of a vision-language model turns into invented objects, plus a single-pass, zero-training allocation rule that existing cache kernels can adopt.