Store the Modality Gap, Don't Quantize It

Systems that search over images and text together almost never keep their embeddings at full precision. A gallery of millions of CLIP vectors is compressed into a few bytes per item and cached in an index, and query-side encoders are increasingly being turned into lookup tables as well. A separate line of work shows that these embeddings carry a large systematic offset between modalities, called the modality gap: image vectors and text vectors sit in two separated clusters even when they describe the same thing. The two facts have not been studied together. This proposal argues that a compressed multimodal cache spends a large share of its bits encoding that offset, which is the same constant for every item of a modality. The proposed fix is to keep one exact gap vector per modality, quantize only what remains, and add the gap back at scoring time with an adjustable strength.

The practical payoff is one compressed index that serves two kinds of task. Mixed-modality search and clustering want the gap removed. Robustness-sensitive uses may want it kept. Today the choice is baked in when the codes are written. The scientific payoff is that the design separates two explanations the literature currently mixes together: the gap as wasted dynamic range, and the gap as protective geometry.

Core Claim

Write each embedding as $x = \mu_m + r$, where $\mu_m$ is the mean embedding of modality $m$ (image or text) estimated once on a calibration set, and $r$ is the residual. We store $\mu_m$ exactly, since it is two vectors for the whole index. We train a separate quantizer $Q_m$ for each modality's residuals. At query time, gallery items are rebuilt as

$$\hat{x}(\alpha) = \frac{\alpha\,\mu_m + Q_m(x-\mu_m)}{\lVert \alpha\,\mu_m + Q_m(x-\mu_m) \rVert},$$

and queries are rebuilt the same way. Here $\alpha \in [0,1]$ is a gap scale chosen per task at scoring time. Setting $\alpha=0$ reproduces post-hoc mean centering. Setting $\alpha=1$ reproduces the original geometry up to quantization error.

The first claim is about compression. At a fixed number of bytes per vector, this scheme gives higher retrieval quality than quantizing raw embeddings. The reason is that scalar and product quantizers size their codebooks to the spread of the data, and the spread is inflated by the shared offset. I0T shows the offset is concentrated in a few dimensions with large modality-specific activation spikes, such as dimension 93 for images in Long-CLIP. We therefore expect the gains to be largest at low bit rates and concentrated in those dimensions.

The second claim is about robustness. Because quantization noise falls on the residual whatever $\alpha$ is, we can test whether the gap's protective effect against noise reported in Is the Modality Gap a Bug or a Feature? survives when the gap is reinjected exactly rather than stored noisily.

Relation to Prior Work

The work that motivates this proposal established that the gap matters for downstream quality. Closing the Modality Gap for Mixed Modality Search subtracts per-modality means (the GR-CLIP method) and reports up to 26 points of NDCG@10 over CLIP on MixBench. Closing the Modality Gap Aligns Group-Wise Semantics retrains with a gap-closing loss and roughly doubles clustering V-Measure on MSCOCO. I0T standardizes embeddings post hoc and reduces the distance between modality centroids from 0.7642 to 0.0102. All three work with full-precision embeddings. None asks what the gap costs once embeddings are compressed into a cache, and none keeps the gap available as a knob after calibration.

Two papers constrain the design by arguing the gap is useful. Mind the Gap traces the gap to independent random initialization of the two encoders and to the contrastive loss. It shows that widening the gap can reduce bias on FairFace and slightly raise CIFAR10 zero-shot accuracy. Is the Modality Gap a Bug or a Feature? proves that an orthogonal gap shields embeddings from Gaussian, quantization and rephrasing noise, with gains of up to 10%. That second result appears to contradict the first claim, because removing the gap before quantizing might remove the shield. We address this directly: the adjustable $\alpha$ exists so the shield can be restored, and the decisive experiment below tests whether it actually is.

The caching side supplies the setting and a secondary target. LightRetriever precomputes a query encoder into a per-token table, averages token vectors at query time, and keeps 95% of nDCG@10 on BeIR while encoding queries over 1000 times faster. Its table is a cache that inherits whatever modality offsets the encoder has, and it was only tested on text. VL-Cache shows that caches inside vision-language models must treat modalities differently: it prunes 90% of the key-value cache with roughly 98% accuracy retained. Its concern, however, is which tokens to keep, not how the geometry of stored vectors should be coded. We take its lesson, that a modality-blind cache policy fails, and apply it to how bits are spent.

Research Program

The core artifact is a compression and scoring layer for multimodal vector indexes that sits on top of frozen CLIP-family and SigLIP encoders. We will build it with scalar quantization at 8 and 4 bits per dimension, and with product quantization (splitting each vector into chunks and replacing each chunk by the nearest of a small learned set of centroids) at 16, 32 and 64 bytes per vector. Each quantizer comes in three variants. The first is fit on raw embeddings. The second is fit after a single global mean is removed, as standard index libraries do. The third uses per-modality means and per-modality codebooks.

The analysis has three parts. First, we decompose quantization error by dimension, to test whether the modality-spike dimensions account for a disproportionate share of the error in the raw variant. Second, we sweep $\alpha$ per task to map how much gap each task prefers. Third, we extend the same split to a LightRetriever-style token table built from a CLIP text tower, where each token vector is coded as offset plus residual. This tests whether table caches benefit the same way.

The main limitation is that a single mean per modality assumes the gap is uniform. The group-wise semantics paper suggests it is not. As a bounded extension, we will try a small number of cluster-specific means, chosen by k-means on the calibration set, and stop if the gain over a single mean is negligible.

Baselines

The naive reference point is uncompressed float32 embeddings, scored both raw and with GR-CLIP centering. This gives the quality ceiling each compressed variant can approach. Its memory cost is reported alongside so the trade-off is visible.

The first compressed baseline is the default practice: quantizing raw embeddings with a quantizer fit on the mixed gallery. This is what a practitioner would deploy today. Beating it means higher NDCG@10 or Recall@K at the same bytes per vector.

The strongest existing method is GR-CLIP from Closing the Modality Gap for Mixed Modality Search, applied before a single shared quantizer. On MixBench it reports up to 26 NDCG@10 points over CLIP and an average 4-point advantage over VLM2Vec, at about 75 times less compute. For two-way retrieval, the I0T post-hoc variant reports Flickr30k image-to-text accuracy of 73.30% and text-to-image accuracy of 76.30%, against 69.60% and 67.10% for CLIP. We found no reported result on Lacuna for either method under quantized storage, so we will measure both ourselves. Every variant gets the same calibration set, the same quantizer training data, the same byte budget and the same hyperparameter search over codebook size. To beat GR-CLIP followed by quantization, our method must either improve quality at equal bytes, mainly through per-modality codebooks, or match it at zero gap while also recovering the noise robustness of the uncentered space when the gap is restored.

Two ablations isolate the key ingredients. The first removes the per-modality codebooks: per-modality centering is kept but a single shared codebook is used. This separates the effect of centering from the effect of specialized codes. The second removes exact gap storage: $\alpha$ is varied by scaling the reconstructions of a raw-quantized index. This separates the benefit of keeping $\mu_m$ exact from the benefit of being able to tune $\alpha$ at all.

Why not just…?

Why not just center with GR-CLIP and then quantize as usual? This is the main threat. If a shared codebook applied after per-modality centering matches our method at every bit rate, the per-modality codebooks add nothing, and the remaining contribution is only the reinjected $\alpha$, which matters only if the robustness effect is real. The first ablation answers this. We will report it as a negative result if centering alone is enough.

Why not just keep separate image and text indexes and calibrate their scores before merging? For mixed-modality search this is a reasonable engineering answer, and it sidesteps the offset during quantization because each index is centered implicitly. It does nothing for clustering or for fusing multimodal documents. The group-wise semantics paper shows those tasks depend on absolute distances across modalities, and GR-CLIP shows that fusing uncalibrated vectors fails. We will include score-calibrated separate indexes as a baseline on MixBench and expect them to be competitive there but not on MSCOCO or MSR-VTT clustering.

Why not just retrain the encoders with a gap-closing loss? The ModGap loss from Closing the Modality Gap Aligns Group-Wise Semantics reduces the gap from 0.47 to 0.03. Doing so requires retraining, fixes a single gap value for all tasks, and, according to the robustness paper, may give up noise protection. We will compress ModGap-style embeddings where a checkpoint is available, to check whether a trained-closed gap quantizes as well as a stored and removed one.

Proposed Experiments

The evaluation covers several task types. Mixed-modality search uses MixBench, scored by NDCG@10. Two-way retrieval uses Flickr30k, scored by image-to-text and text-to-image accuracy. Clustering uses MSCOCO and trimodal MSR-VTT, scored by V-Measure. Each is plotted as quality against bytes per vector, so the result is a curve per method rather than a single number. Noise robustness follows the protocol of the robustness paper: Gaussian noise, plus quantization noise applied at the residual level, measured on the compressed indexes.

The first decisive check tests the compression claim. At 4-bit scalar quantization and 16-byte product quantization, the per-modality offset-plus-residual scheme should beat raw quantization on Flickr30k and MixBench. Error should concentrate in the I0T spike dimensions under raw quantization and disappear from them under our scheme. If neither holds, the wasted-range explanation is wrong.

The second decisive check settles the conflict between the two schools. We compare three compressed variants under added noise: raw quantization at $\alpha=1$, our scheme at $\alpha=1$, and our scheme at $\alpha=0$. Suppose ours at $\alpha=1$ keeps the robustness of the raw index while matching the quality of $\alpha=0$ on mixed search after retuning. Then the gap is protective geometry that is better stored than coded. Suppose instead the robustness advantage vanishes once quantization error is controlled. Then the reported protection was largely a bit-allocation effect.

The third check probes the cache side. A CLIP text tower converted into a LightRetriever-style token table is used as the query encoder on MixBench, with and without offset-plus-residual coding of its table. This tests whether table-based query caches suffer the same wasted range as gallery indexes.