Low-Bit Quantization Rounds Vision-Language Models Back Toward Their Text Parents

Most vision-language models are built in sequence. A language model is first trained on trillions of text tokens. Then, in a much shorter second stage, it is taught to accept image features from a separately trained vision encoder. This note proposes a simple, testable reason why such models lose their visual reasoning first when their weights are compressed to 2 or 3 bits. The hypothesis is that everything the language backbone learned about images during the short second stage lives in a small weight change, the "multimodal delta," laid on top of the text model. That change is small compared with the rounding step of a 4-level or 8-level grid. Low-bit quantization, which snaps each weight to the nearest of a few allowed values, may therefore erase the delta and push the model back toward the text-only model it started from. The model would still write fluent sentences but would no longer look at the image.

If this is right, it explains two separate observations with one mechanism. The first is the "visual reasoning cliff," where LLaVA-Next-8B keeps grammatical fluency at 2 bits yet fails on diagrams and charts (An empirical study of LLaMA3 quantization). The second is the tendency of vision-language models to trust text over contradicting images (Words or Vision). The hypothesis also suggests a fix that current methods do not use. The quantizer should be anchored to the text parent, and the delta should be stored separately. Two common approaches both miss this: treating the finished model as one block of weights, and giving the extra bits to whichever weights look most important.

Core Claim

Write the backbone weights of a sequentially trained vision-language model as $W = W_0 + \Delta$. Here $W_0$ is the public text-only parent, for example Vicuna-1.5-7B for LLaVA-1.5-7B or LLaMA3-8B-Instruct for LLaVA-NeXT-8B, and $\Delta$ is what the vision-language stage added. The claim has two parts, and each can be falsified. First, at 3 bits and below, the fraction of $\Delta$ that survives quantization collapses before overall quality does. The loss of "image-dependent" accuracy tracks that survival fraction rather than overall weight error. Second, a quantizer that rounds $W_0$ and stores $\Delta$ in a compact separate form, at the same total bits per weight, recovers visual reasoning that saliency-based mixed precision and modality-weighted calibration do not.

To measure survival, define for each layer

$$ s = \frac{\langle Q(W) - W_0,\ \Delta \rangle}{\lVert \Delta \rVert^2}, $$

where $Q(W)$ is the quantized weight matrix and $\langle\cdot,\cdot\rangle$ is the elementwise inner product. A value of $s = 1$ means the quantized model keeps the full multimodal change. A value of $s = 0$ means that, along the direction of $\Delta$, the quantized model has reverted to the text parent.

How this sits in prior work

Three quantization papers motivate the gap. MBQ finds that language tokens carry more than 10× larger loss gradients than vision tokens, and it reweights calibration to compensate. VEQ measures a 22.4× gap in mixture-of-experts vision-language models and adds modality- and expert-aware Hessian weighting. Q-VLM tunes the vision encoder so that its outputs quantize more easily at 4 bits. All three treat the trained weights as a single object and never ask where in weight space the visual skill sits. We ask exactly that. In our framing, the large text-to-vision gradient ratio is a symptom: the visual pathway is a small correction on a model shaped overwhelmingly by text.

Two further papers constrain the design. Low-Bit Quantization Favors Undertrained LLMs shows that quantization damage grows with training tokens, roughly as $D^{0.53}$. This implies that the heavily trained parent $W_0$ is the fragile part. It also implies that simply lengthening the vision-language stage to "grow" $\Delta$ may add fragility rather than remove it, so we test that directly instead of assuming it. The LoRA paradox reported by Huang et al., where fine-tuning a quantized LLaMA3 made it worse, contrasts with QSLAW, which reaches 91.04% on ScienceQA with quantization-aware scale learning against 86.96% for QLoRA. Together they warn that repair by fine-tuning after quantization is unreliable. That is why our main intervention involves no training.

The sequential-training literature explains why a small $\Delta$ is plausible in the first place. Bogin et al. match baselines using 5% of the paired data when starting from strong unimodal models. Should VLMs be Pre-trained with Image Data? contrasts the standard text-first pipeline with introducing images earlier. Georgiou et al. (link) describe how the text modality dominates optimization. Both lines point to a backbone that barely moves during multimodal training, and we test whether it in fact barely moves. We borrow the rotation technique from QuIP as a strong quantizer to build on. We use the saliency-based mixed precision of CherryQ as the main budget-matched competitor.

Research Program

The first part is measurement. We take several vision-language models whose text parents are public and compute $\Delta$ for every backbone matrix. We then quantize each model to 4, 3, and 2 bits with GPTQ, AWQ, and QuIP. Throughout, the vision encoder and projector stay in 16-bit, so that any effect comes from the backbone alone. For each setting we record $s$ per layer, along with the ratio of $\lVert\Delta\rVert$ to the rounding step. We use models whose vision-language stages differ in length. LLaVA-1.5 has a short stage with a fully fine-tuned backbone. LLaVA-NeXT-8B sits on a 15T-token parent. Qwen2-VL, where the parent is available, has a much longer multimodal stage. This variation lets us check whether a longer stage produces a larger $\Delta$, and whether a larger $\Delta$ survives better.

The second part is the behavioral link. For every benchmark item we score the model with and without the image. We define image-dependent gain as the difference in accuracy between the two conditions. The prediction is that image-dependent gain falls faster than overall accuracy as bits drop, and that the Text Preference Ratio from Words or Vision rises. In other words, quantization should amplify blind faith in text.

The third part is the fix, which we call parent-anchored quantization. We quantize $W_0$ with QuIP at $b$ bits. We then fit $\Delta$ with a low-rank factorization, adding a small sparse residual if needed, and store it at 4 bits. The rank is chosen so that total bits per weight equal those of the competitor it is compared against. At inference time the stored factors are added back onto the dequantized parent weights. A single parent quantization can then serve several vision-language models built on the same parent, which has practical value of its own.

The fourth part, carried out only if the first three succeed, varies the vision-language stage itself at LLaVA-1.5-7B scale, where that stage is cheap. We compare a frozen backbone, a fully fine-tuned backbone, and a backbone with a 2× longer schedule. We then measure $s$ and 3-bit image-dependent gain for each. This connects the mechanism back to curriculum design.

Baselines

The naive reference point is round-to-nearest uniform quantization of the full model at each bit width. It shows the size of the cliff, and Das et al. report that uniform methods need 6.0–10.0 bits per weight to retain performance, against 3.5–4.5 for GPTQ and AWQ.

The strongest existing method for vision-language models specifically is modality-balanced calibration. We run MBQ (MBQ) on the dense LLaVA models and VEQ (VEQ) where a mixture-of-experts model is included. Both use the same 128 ShareGPT4V/COCO calibration pairs that those papers use. The strongest reported number at 3 bits is VEQ's Kimi-VL W3A16 average accuracy of 65.41% across MMMU, MMBench, ScienceQA, TextVQA and RealWorldQA, 2.04% above AWQ, with a remaining 9–11% gap to BF16. MBQ reports gains of up to 4.4% at W3A16 over AWQ. The file gives no absolute number for LLaVA-1.5-7B at 3 bits. We therefore re-run MBQ ourselves, and we count a win as closing a significant share of that model's gap to 16-bit on image-dependent gain at matched bits per weight.

The strongest general-purpose competitor is CherryQ-style mixed precision (Cherry on Top). It keeps the most impact-heavy weights in 16-bit and reports a 2-bit LLaMA2-7B perplexity of 9.55, against more than $10^5$ for AWQ. We give it exactly the extra bits that parent anchoring spends on $\Delta$. We also run QuIP alone as the rotation-only reference.

The ablation that isolates our key ingredient spends the same low-rank budget on the quantization residual $W - Q(W)$ instead of on $\Delta$. This is standard low-rank error correction, and it differs from our method only in what the extra bits are anchored to. A second ablation replaces $\Delta$ with a random matrix of equal norm. It tests whether the specific direction of $\Delta$ matters, rather than the magnitude of the perturbation. Every method gets the same calibration data, the same frozen encoder and projector, and the same total bits per weight counted with all scales and factors included.

Why not just…?

Why not just use modality-balanced calibration like MBQ or VEQ? Reweighting calibration changes how rounding error is spread across output channels. It does not change the fact that a weight's multimodal change can be smaller than half a grid step, and in that case no choice of rounding keeps it. VEQ's own persistent 9–11% gap at 3 bits is consistent with this ceiling. This alternative is a real threat, however, and our measurement settles it. If MBQ leaves $s$ near 1 at 3 bits, our hypothesis is wrong for that regime.

Why not just keep the most sensitive weights in high precision, as CherryQ does? Saliency scores are computed from the loss on calibration data dominated by text-sized gradients, so they will tend to protect the parent's text circuits. The multimodal delta is also probably spread thin across many weights rather than concentrated in a few. The budget-matched comparison, together with a count of how much of $\Delta$ falls inside CherryQ's protected set, will show which account holds.

Why not just fine-tune the quantized model on vision-language data afterward? The literature disagrees on whether this helps: the LoRA paradox in Huang et al. cuts against it, while QSLAW supports it. It also costs training compute that our method avoids. We include a QSLAW-style scale-learning arm on LLaVA-1.5-7B at matched bits. If it matches parent anchoring on image-dependent gain, the practical argument for our method narrows to reuse and training-free deployment, and we will say so.

Why not just stay at 4 bits? At 4 bits LLaVA-Next-8B loses less than 2% (Huang et al.). If $s$ stays high at 4 bits and only 3- and 2-bit budgets matter, the scope of this work is exactly those budgets, which is where the memory savings are.

Proposed Experiments

We evaluate on ScienceQA split by whether the question has image context, TextVQA, MMMU, MMBench, AI2D, ChartQA and DocVQA. The contradiction suite of Words or Vision supplies the Text Preference Ratio. That paper reports that LLaVA-NeXT-7B reaches 28.69% accuracy under text corruption at full precision, and we expect this number to fall further under quantization. For text-only ability we use WikiText-2 perplexity and GSM8K, to confirm that parent anchoring does not trade away language quality. The metrics are overall accuracy, image-dependent gain, Text Preference Ratio, and $s$ per layer.

The first decisive check is a swap test. We compare $Q(W_0) + \Delta$ in 16-bit against $Q(W)$ at the same parent bit width. If adding the exact delta back does not restore most of the lost image-dependent gain, the mechanism is wrong. In that case the damage lies in the parent's own circuits, which would point toward Ouyang et al.-style saturation rather than delta erasure.

The second decisive check is correlation across all models, methods and bit widths. Image-dependent gain should track $s$ more closely than it tracks overall weight reconstruction error.

The third decisive check is the budget-matched head-to-head against MBQ, CherryQ, and residual low-rank correction at 3 and 2 bits. The equal-norm random-delta control should show no recovery.

A negative result would also be informative. If $\Delta$ turns out to be large relative to the grid and survives quantization, the visual cliff must come from somewhere else, such as outliers at the vision-text interface. That would redirect effort toward the connector-side explanations raised by QSLAW.