When a vision-language model is compressed to 2-bit weights, it keeps producing fluent sentences but loses the ability to reason about the image. Diagram and chart questions collapse into repetitive or invalid answers (An empirical study of LLaMA3 quantization: from LLMs to MLLMs). The standard repair adds a small full-precision low-rank "patch" to each quantized weight matrix to cancel its rounding error. That repair stops working at 2 bits, because the error no longer concentrates in a few directions (LQER: Low-Rank Quantization Error Reconstruction for LLMs). We propose to stop trying to repair the whole weight error. Instead, the patch should repair only the part of the error that damages how text looks at the image.
The multimodal fusion literature gives a reason to think this part is small. When fusion is done well, cross-modal interactions sit in a low-dimensional subspace. Low-rank fusion models exploit this with modality-specific factors (Efficient Low-rank Multimodal Fusion With Modality-Specific Factors), and in vision-language models the matrix of text-query/visual-key attention scores is itself low-rank (Vision Token Pruning via Query--Vision Interaction Decomposition). If the visual reasoning cliff comes from quantization scrambling the few dominant modes of that matrix, then a correction aimed at those modes needs a rank set by the interaction rank, not by the error spectrum. That could get around the 2-bit barrier for a small fraction of the memory that mixed precision would cost.
Core Claim
In a vision-language model, write $M = X_t W_Q W_K^\top X_v^\top$ for the matrix of attention scores between text tokens and visual tokens within one head. Here $X_t$ and $X_v$ are the text and visual hidden states entering the layer, and $W_Q, W_K$ are that head's query and key projections. Quantizing both projections with errors $E_Q, E_K$ perturbs the scores by
$$\Delta M = X_t\left(E_Q W_K^\top + W_Q E_K^\top + E_Q E_K^\top\right)X_v^\top .$$
This is bilinear. Errors in the two factors multiply, which is the same compounding that affects Hadamard-product fusion in LMF-style models. Existing post-training quantizers minimize each layer's output error separately and never see this product.
Let $U_r, V_r$ be the top-$r$ left and right singular vectors of the full-precision $M$ on a given calibration sample. We fit rank-$k$ full-precision patches $A_Q B_Q$ and $A_K B_K$, added to the quantized projections, by minimizing the expected projected interaction error:
$$\min_{\text{rank-}k\ \text{patches}} \ \mathbb{E}_{s}\left\| U_r(s)^\top \, \Delta M_{\text{patched}}(s)\, V_r(s) \right\|_F^2 ,$$
where the expectation runs over a mixed image-text calibration set. Because the objective only asks the patch to fix an $r\times r$ block per sample, a rank near $r$ should suffice. That holds whether or not the raw weight error has decaying singular values.
The claim to test has three parts. First, the 2-bit visual reasoning cliff is largely explained by loss of alignment between the full-precision and quantized dominant interaction subspaces. Second, restoring that subspace recovers most of the lost multimodal accuracy. Third, a patch fit this way beats both weight-error patches and higher-precision Q/K projections at equal total memory.
Relation to Prior Work
Four papers sit closest to this idea, and each leaves a specific opening.
- LQER approximates the full weight error with a low-rank patch. Its activation-scaled variant L²QER keeps text LLMs within 0.3% of FP16 at W4A8 on LLaMA-2-70B. It hits a "2-bit wall" and was never evaluated on multimodal models. We keep its hardware-friendly patch form but change what the patch is fit to preserve.
- MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression jointly searches rank and bit-width per linear layer for ViT and BERT. It explicitly leaves vision-language models as future work, and its objective is still per-layer reconstruction rather than a cross-layer bilinear quantity.
- LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models motivates the multimodal focus. It shows image activations carry higher entropy and assigns whole layers 4 or ultra-low bits by an entropy heuristic. It uses no low-rank structure.
- LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models moves low-rank approximation of multimodal weights into the frequency domain. It targets moderate compression, not the sub-3-bit cliff.
Other work constrains the design. Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models finds that loss-impact scores predict quantization sensitivity better than activation magnitude. That argues for a task-relevant objective like ours over magnitude heuristics. It also means sparse FP16 outliers are a competitor we must beat.
QViD supplies the key measurement: text-vision score matrices are low-rank, and their top singular vectors can be computed cheaply with a QR trick. QViD uses this only for token pruning at full precision. We use the same structure as the target that quantization must preserve.
Research Program
Diagnosis. On LLaVA-1.5-7B and LLaVA-Next-8B, we quantize with GPTQ, AWQ and QuIP at 4, 3 and 2 bits. For each layer and head, we measure the principal angles between the full-precision and quantized top-$r$ interaction subspaces on held-out image-question pairs. We then ask whether subspace misalignment, aggregated over early and middle layers, predicts per-benchmark accuracy loss on AI2D and ChartQA better than two controls: plain weight reconstruction error, and LUQ's activation entropy.
An oracle intervention settles whether the Q/K interaction is the bottleneck. We restore only $W_Q$ and $W_K$ to FP16 in an otherwise 2-bit model and measure recovery. We do the same for value/output projections and for MLPs.
Method. For each attention layer, we fit the projected-interaction patches. With the other factor held fixed, the objective is a weighted linear least-squares problem in each patch. We alternate between the query and key patches for a few rounds, and solve each step as a reduced-rank regression after whitening the calibration statistics.
Rotary position encodings rotate queries and keys per position, so $M$ is not a pure function of $W_QW_K^\top$. We therefore compute $\Delta M$ from the actual rotated calibration activations rather than from the weight product. The problem remains linear in each patch.
The rank $k$ is tied to the measured interaction rank $r$ of each layer, which also addresses the manual rank selection that Dual-LMF flags as open (Dual Low-Rank Multimodal Fusion). Calibration uses 1:1 mixed image-text samples, following LUQ's finding that this helps.
Controlled testbed. Before the large-model work, we quantize the modality-specific factors of an LMF model on CMU-MOSI to 3 and 2 bits. Here the bilinear, Hadamard-product fusion is explicit and small. We fit the same subspace-targeted patch to the fused representation's dominant modes. This checks the compounding argument and the rank claim in a setting where exact error propagation can be computed.
Memory accounting. All comparisons charge the patch against the bit budget. A rank-32 FP16 patch on a 4096×4096 matrix costs $2\cdot 32\cdot 16/4096 = 0.25$ bits per weight of that matrix. Query and key projections are roughly one sixth of a LLaMA-7B-style layer's weights, so patching both adds about 0.04 bits to the model average. Keeping them at 4 instead of 2 bits costs about 0.33.
Baselines
The naive reference points are the FP16 model and uniform 2-bit GPTQ/AWQ/QuIP with no correction. For the latter, the file reports only a qualitative collapse on AI2D and ChartQA, with no numeric value (An empirical study of LLaMA3 quantization: from LLMs to MLLMs). We will measure these numbers ourselves under identical prompts and decoding. QuIP's rotation step, which spreads outlier energy across coordinates before rounding (QuIP: 2-Bit Quantization of Large Language Models With Guarantees), is the base quantizer onto which all patches are added, so every comparison shares the same 2-bit core.
The strongest existing method for this question is LUQ. It reports over 90% retention of FP16 performance, with under 10% loss on MME, at about 2.5–2.7 average bits on LLaVA-1.5-7B and Qwen-2.5-VL-7B, and a 40% memory reduction over 4-bit on LLaVA-1.5. We run LUQ with its own entropy allocation and matched calibration data at the same average bits our method uses, including patch cost. Beating it means higher accuracy at equal or lower average bits on AI2D and ChartQA, where the cliff lives, without losing ground on MME, TextVQA and POPE.
The most direct competitor for the mechanism is L²QER applied to vision-language models with the same mixed calibration set and the same patch ranks. No reported result was found on Lacuna for LQER or L²QER on any multimodal model; its only reported figure is the W4A8 text result above.
Two ablations isolate the key ingredient.
- Objective ablation. We hold rank, calibration data and base quantizer fixed and vary only the patch objective: weight error (LQER), per-layer output error, full unprojected interaction error $\|\Delta M\|_F$, our top-$r$ projected objective, and a random $r$-dimensional subspace as a control. If the projected objective does not beat the unprojected one, the subspace idea contributes nothing beyond "fit the product."
- Outlier comparison. We compare against CherryQ-style FP16 preservation of the highest-impact Q/K parameters at matched storage. This tests whether sparse outliers, rather than a low-rank subspace, carry the cross-modal signal.
Why not just…?
Why not just keep the query and key projections at 4 bits and quantize everything else to 2? This is the real threat, and it is LUQ's logic applied at the matrix level. If the oracle experiment shows that FP16 Q/K recovers most accuracy, 4-bit Q/K may too. It costs roughly 0.33 extra average bits against about 0.04 for rank-32 patches, so the fair test is the memory-matched curve. We run 4-bit Q/K, 3-bit Q/K, and patched 2-bit Q/K, each paired with enough extra budget elsewhere to equalize average bits. If 4-bit Q/K matches our method at equal memory, we will report that the simpler scheme suffices and the subspace objective is unnecessary.
Why not just apply L²QER with multimodal calibration data? LQER's own analysis says the 2-bit weight error lacks fast singular value decay, so its rank must grow until the savings vanish. Multimodal calibration changes the activation scaling but not the objective, which still asks for the whole error. The objective ablation measures exactly this gap at equal rank. The rank-sweep experiment shows whether L²QER needs a much larger $k$ to reach the same AI2D and ChartQA accuracy.
Why not just fine-tune the 2-bit model with QLoRA? The LLaMA3 study found that LoRA fine-tuning of heavily quantized models on small instruction sets lowered performance below the uncorrected quantized model, which it attributes to forgetting. Fine-tuning also needs far more compute than a calibration pass: LQER quantizes LLaMA-33B in about 1.2 hours on one A100. We include a QLoRA run on the 2-bit model with an instruction set matched in compute to a generous budget. This shows whether that negative result carries over to multimodal reasoning.
Proposed Experiments and Decisive Checks
The first decisive check is the oracle. If restoring $W_Q, W_K$ to FP16 in a 2-bit LLaVA-1.5-7B recovers less than about half of the AI2D and ChartQA gap, the cliff does not originate in the query-key interaction. In that case the method would be redirected to the value/output path, where the same projected objective applies to the attention-weighted visual values, or abandoned.
The second check is predictive. Subspace misalignment should rank the 4/3/2-bit configurations and the three quantizers by AI2D/ChartQA loss better than weight error or activation entropy do.
The third check is the memory-matched accuracy curve. On LLaVA-1.5-7B, Qwen-2.5-VL-7B and LLaVA-Next-8B, we sweep average bits from about 2.1 to 3. We evaluate AI2D, ChartQA, MME, MMBench, TextVQA, POPE and Object HalBench, plus WikiText-2 perplexity to confirm that language ability is not traded away. The method must dominate LUQ, L²QER and 4-bit Q/K on the cliff benchmarks at matched bits.
Finally, the CMU-MOSI testbed should show the same pattern at small scale. LMF's reported full-precision MAE of 0.912 serves as the reference. A subspace-targeted patch on 2-bit modality factors should close more of the MAE gap than a weight-error patch of equal rank. The required rank should track the fused representation's measured interaction rank rather than the factor error spectrum.
References
- An empirical study of LLaMA3 quantization: from LLMs to MLLMs
- LQER: Low-Rank Quantization Error Reconstruction for LLMs
- Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
- Vision Token Pruning via Query--Vision Interaction Decomposition
- MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression
- LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models
- LLaVA-FA: Learning Fourier Approximation for Compressing Large Multimodal Models
- Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
- Dual Low-Rank Multimodal Fusion
- QuIP: 2-Bit Quantization of Large Language Models With Guarantees