Modality-Centered Compression: Stop Spending the Error Budget on the Modality Gap

Vision-language models place image-derived and text-derived vectors in two separate clusters, a well-documented "modality gap." When such a model is compressed, whether by rounding weights to few bits (quantization), zeroing some of them (pruning), or replacing weight matrices with thinner factored ones (low-rank decomposition), the compressor is calibrated on a batch that mixes both clusters. Each layer's error then includes a large, systematic term: the compressed weights applied to the offset between the two cluster centers. That term is deterministic and can be computed in advance. We propose to remove it exactly. Each linear layer is compressed so that it acts on inputs with their own modality's mean subtracted. The discarded mean contribution is then restored through a per-modality correction vector, which at inference costs one extra bias add per token. The compressor's whole error budget goes to the within-modality variation that carries meaning, and the geometric gap between modalities survives compression unchanged by construction.

This matters for two reasons. In practice, compound pipelines such as low-rank plus 4-bit or 2:4 sparsity plus low-rank are the most promising route to large memory savings, but they were built and tuned on text-only models. When several compressors are stacked, a mean-offset error left by one becomes a shifted input distribution for the next. The second reason is scientific. The literature disagrees on whether the modality gap is a defect to be removed or a buffer that protects against noise. Nobody has measured what real compression does to it, and a method that holds the gap fixed turns that disagreement into a clean downstream choice instead of an uncontrolled side effect.

Core Claim

Take a linear layer with weight $W$, compressed estimate $\hat W$, and an input token $x$ from modality $m$ (image or text) whose calibration mean is $\mu_m$. Standard calibration minimizes the error $(W-\hat W)x$ averaged over all tokens. We rewrite the layer output as

$$ Wx \;=\; W(x-\mu_m) + W\mu_m \;\approx\; \hat W(x-\mu_m) + W\mu_m \;=\; \hat W x + c_m,\qquad c_m = (W-\hat W)\mu_m . $$

We fit $\hat W$ to minimize error on the centered inputs $x-\mu_m$ only. The correction $c_m$ is stored in 16-bit precision and added to every token of modality $m$. In practice, every second-moment matrix that current compressors rely on is replaced by a sum of within-modality covariances. That includes the Hessian proxy $XX^\top$ used by GPTQ-style quantizers, the whitening matrix in SVD-LLM V2, and the activation statistics in 3BASiL's ADMM. The mean term is not approximated; it is removed from the problem.

The central prediction is quantitative. The benefit at a given layer should grow with the ratio of between-modality mean separation to within-modality spread, because that ratio sets how much of the standard objective the mean term consumes. The benefit should also compound when several compressors are stacked, because each one otherwise hands the next a mean-shifted input.

Relation to Prior Work

Several papers motivate the proposal. Mind the Gap shows the gap exists across contrastive models and arises from the architecture and the loss. Towards understanding the modality gap in CLIP reports that a linear classifier separates the two modalities in CLIP with 100% accuracy, so the offset is large and structured. Is the Modality Gap a Bug or a Feature? proves the gap vector is orthogonal to both modalities' subspaces and acts as a noise buffer. That work, however, applies quantization noise synthetically to output embeddings rather than compressing any weights. We compress weights for real and ask whether the buffer survives.

On the "bug" side, two works show that downstream tasks sensitive to distance suffer from the gap. Closing the Modality Gap Aligns Group-Wise Semantics shows this for clustering and Closing the Modality Gap for Mixed Modality Search (GR-CLIP) for mixed-modality search. The first paper also establishes that ordinary retrieval ranks are insensitive to the gap. That result shapes our evaluation: retrieval recall alone could hide gap drift caused by compression, so we must also report clustering and mixed-modality search.

Two further works constrain the design. Extending Multi-modal Contrastive Representations proves that a single transformation layer cannot both resolve the gap and respect an external embedding geometry. We therefore do not try to close the gap inside compressed weights; we preserve it and leave closing it to a separate post-hoc step. SQAP-VLA shows that naively stacking 4-bit quantization with token pruning fails because quantization shifts attention. Its evidence that compounding errors are real motivates our focus on stacked pipelines.

The compression machinery we reuse comes from four sources. The joint sparse-plus-low-rank decomposition is 3BASiL. The SVD-then-quantize pipeline is SVD-LLM V2. Modality-balanced quantization is MBQ, which weights calibration error by each token's gradient sensitivity after finding language tokens over 10× more sensitive than vision tokens. VEQ is a mixture-of-experts analogue that reports a 22.4× gradient ratio.

MBQ and VEQ reweight how much each modality's error counts. Neither removes the systematic mean-offset component, and neither measures latent geometry. Our centering is complementary to that reweighting and can be combined with it. To our knowledge from this file, no work applies modality-conditional centering inside a compressor's calibration, although single-population bias correction is a natural relative that we include as an ablation.

Research Program

The first stage is diagnostic. On a dual-encoder CLIP model and on decoder VLMs where image and text tokens share weights (LLaVA-onevision-7B and Qwen2-VL, following MBQ), we record per-layer modality means and covariances. We use the 128 ShareGPT4V-COCO image-caption pairs that MBQ calibrates on. For each layer and each compressor we decompose the standard calibration error into three parts: the mean term $(W-\hat W)\mu_m$, the per-token component along the global gap direction, and the within-modality remainder. We also track the embedding-level gap (centroid distance and angle) before and after compression. No prior work in the file reports this measurement. It tells us directly whether uncorrected compression shrinks the gap, enlarges it, or rotates it.

The second stage builds the method. We implement modality-centered variants of GPTQ-style 3-bit and 4-bit weight quantization, of SVD-LLM V2's two-round SVD with 4-bit quantization, and of 3BASiL's 2:4-sparse plus rank-64 decomposition. Each also gets a stacked version, for example 2:4 sparsity, then low rank, then 4-bit. In each case the only change is that calibration statistics are computed on modality-centered inputs and the correction vectors $c_m$ are folded in as per-modality biases. In a VLM the image token positions are known at inference, so the correction vector can be selected with no routing overhead.

The third stage connects compression to the bug-versus-feature debate. Because centered compression holds the modality means fixed, we can compare three settings on the same compressed model: gap kept as is, gap removed post hoc with GR-CLIP-style mean subtraction, and gap enlarged along the orthogonal direction as Chowers et al. propose. That comparison shows which choice helps which task once the model is compressed.

Baselines

The do-nothing reference is the uncompressed 16-bit model, together with naive round-to-nearest quantization. These bound the achievable range.

The strongest existing competitor for decoder VLMs is MBQ. It is the only method in the file designed around modality asymmetry under quantization, and it reports accuracy improvements of up to 4.4% at W3A16 and up to 11.6% at W4A8 across MMMU, TextVQA, OCRBench and SEED-Bench, with under 1.1% loss at 72B scale (MBQ). The file gives no per-benchmark MBQ figures, so we will rerun MBQ ourselves. MBQ and our method will use the same 128 calibration pairs, the same bit-widths and group sizes, and the same model checkpoints, and each method gets the same hyperparameter sweep. Our method counts as beating MBQ if centered-MBQ, meaning MBQ's gradient weighting plus our centering, improves average accuracy over plain MBQ at W3A16 across three calibration seeds, by more than the seed-to-seed variation.

For compound compression, the baselines are SVD-LLM V2 and 3BASiL-TM. On WikiText-2, 3BASiL-TM reports reducing the perplexity gap to dense Llama-3-8B by over 30% at 2:4 plus rank 64 (3BASiL). SVD-LLM V2 reports 69% lower perplexity than BiLLM when combined with 2-bit quantization (SVD-LLM V2). No reported result was found on Lacuna for either method on a vision-language model. We will port both to the VLMs unchanged and compare centered against uncentered versions at equal memory footprint and equal compression runtime.

For dual-encoder CLIP, the gap-aware post-hoc reference is GR-CLIP applied to the compressed model. It reports up to 26 points NDCG@10 over CLIP on MixBench (Closing the Modality Gap for Mixed Modality Search). The clustering reference is ModGap's V-Measure of 23.63 on MSCOCO against roughly 13.0 for CLIP (Closing the Modality Gap Aligns Group-Wise Semantics). Our target here is not to exceed ModGap, since ModGap retrains the model. The target is that compressed models lose less of their uncompressed clustering and mixed-search score.

The ablation that isolates the key ingredient is centering with a single global mean instead of per-modality means. This is ordinary bias correction, and it is the key control. If it matches per-modality centering, the cross-modal story is wrong and only generic bias correction matters.

Why not just…?

Why not just compress with a standard method and then subtract modality means afterward, as GR-CLIP does? This is the most serious alternative for dual encoders, because it fixes the final-embedding mean shift for free. It does nothing, though, about mean-offset errors injected at intermediate layers. Those errors propagate through nonlinearities and do not reduce to a shift at the output. In decoder VLMs there is no final embedding to recenter at all. The experiments settle the question by running "standard compression plus post-hoc recentering" against "centered compression" at matched memory. If post-hoc recentering closes most of the gap between them on CLIP, we will report that honestly and restrict the claim to shared-weight decoders and stacked pipelines.

Why not just use MBQ's gradient-weighted calibration? MBQ changes how much each modality's error counts but leaves the deterministic mean term inside the objective. The two mechanisms are orthogonal, which is why our main comparison is MBQ against centered-MBQ rather than a replacement. If centering adds nothing on top of MBQ, the mean term is not the bottleneck, and our diagnostic stage will already have shown this through a small mean-term share of the error.

Why not just apply ordinary single-mean bias correction? Single-mean correction removes the average of the mixed image and text population. When the two modalities sit in separate clusters, that global mean lies between them and fits neither, leaving roughly half the offset in each modality's error. This is an empirical claim, not a certainty, and the global-mean ablation tests it directly. We expect its gap from our method to track the measured between-to-within spread ratio.

Why not just keep vision layers at higher precision or rank? Mixed allocation spends extra memory. Centering costs one vector per layer per modality. We will include a memory-matched mixed-precision baseline to show whether the memory is better spent on extra bits or on centering.

Proposed Experiments

On decoder VLMs (LLaVA-onevision-7B and Qwen2-VL), we evaluate MMMU, TextVQA, OCRBench and SEED-Bench under W3A16, W4A8, SVD with 4-bit quantization at 20–40% rank reduction, and 2:4 sparsity plus rank 64 plus 4-bit. Each setting runs with and without centering, combined with MBQ weighting, over three calibration seeds.

On CLIP we compress the image and text encoders with the same pipelines. We then report MSCOCO and Flickr30k retrieval recall, MixBench NDCG@10, MSCOCO clustering V-Measure, ImageNet zero-shot accuracy, and robustness to Gaussian noise following Chowers et al. Clustering and mixed search are included deliberately, because retrieval recall alone is insensitive to the gap and could hide drift.

Three checks decide whether the idea works. First, across all layers and compressors, the per-layer accuracy or reconstruction benefit of centering should correlate positively with the measured between-to-within modality spread ratio; no correlation would falsify the mechanism. Second, per-modality centering should beat global-mean centering, and the margin should be largest in stacked pipelines, where uncorrected offsets compound. Third, embedding-level gap drift should be near zero for centered compression and measurably nonzero for uncentered compression. With drift controlled, noise robustness should be higher when the gap is kept or enlarged, while clustering and mixed search should be better after post-hoc gap removal. That pattern would reconcile the bug-versus-feature debate for compressed models: the gap should be preserved during compression, and closing it should be a separate downstream decision.