Does the Modality Gap Depend on How Common a Concept Is?

Image-text models such as CLIP embed pictures and captions into one shared space, yet the two kinds of embedding sit in separate regions of that space. This separation is called the modality gap. Every current fix treats it as a single constant offset: it computes the average image vector and the average text vector and subtracts the difference. Meanwhile, work on word statistics shows that how often a word or concept appears in training data shapes its geometry. A concept's frequency changes how spread out its representations are, how long its vectors are, and how the model lays out numbers along a line.

This note asks whether the modality gap also depends on concept frequency, following the power-law pattern (Zipf's law) that governs how often concepts appear. If it does, the offset for a concept could be predicted from its frequency alone. Rare concepts could then be corrected even when no paired image-text examples of them are available for calibration. Global centering cannot do this, and neither can per-concept centering.

Core Claim

The claim is that the per-concept modality gap is heterogeneous, and that its variation is predictable from log concept frequency in the pretraining data. This allows a frequency-conditioned gap correction. The correction is fit on head concepts, where calibration pairs are plentiful, and extrapolated to tail concepts, where they are scarce or absent. The regime where it should matter most is long-tail entity retrieval and mixed-modality search.

The obvious combination of the two directions would be to apply Zipfian Whitening to CLIP. That paper computes a frequency-weighted mean and a frequency-weighted decorrelation for text embeddings, and it tests only on text. I argue this obvious transfer is mostly already done, in disguise. GR-CLIP subtracts the arithmetic mean over a calibration set of instances. An average over instances is already weighted by how often each concept occurs, which is exactly the "token, not type" weighting that Zipfian Whitening argues for. Zipfian centering of CLIP would therefore largely reproduce GR-CLIP.

What neither line of work tests is whether one global offset is correct in the first place. The research file names this "homogeneous gap" assumption as unexamined across Mind the Gap, GR-CLIP, I0T, and the robustness analysis. The question addressed here is that assumption, and the prediction that follows from dropping it.

To make it precise, let $\delta_c = \bar{u}_c - \bar{t}_c$ be the concept-level gap. Here $\bar{u}_c$ and $\bar{t}_c$ are the mean normalized image and text embeddings for concept $c$. The model is

$$\delta_c \approx \delta_0 + \phi(\log f_c)\, D,$$

where $f_c$ is the concept's pretraining frequency, $\phi$ maps log-frequency to a few smooth basis features ($1$, $\log f$, $(\log f)^2$), and $D$ is a small matrix of learned direction coefficients. Setting $D=0$ recovers global centering.

Why frequency should matter

The motivating evidence comes from unimodal work, which shows that frequency shapes geometry. A New Formulation of Zipf's Meaning-Frequency Law measures how widely a word's contextual embeddings spread and finds that the spread scales as a power law of frequency ($R^2$ 0.84–0.91). Pretraining Numerical Frequency and Number-Line shows that the rate at which number frequencies decay in a corpus predicts how compressed the model's internal number line is ($\rho_s = 0.84$).

Two gap mechanisms also suggest frequency effects:

  • The cone effect. Deep networks squeeze embeddings into a narrow cone at initialization, and each modality's encoder has its own cone (Mind the Gap). Rarely trained concepts may sit closer to these starting cones.
  • The contrastive minimum. The standard contrastive loss has a stable local minimum that keeps the modalities apart (It's Not a Modality Gap, ModGap). Frequently seen concepts receive more gradient pressure toward that equilibrium.

These mechanisms predict frequency dependence but not its sign. The experiments decide the sign.

Two findings constrain the design:

  • Imbalance breaks existing gap-closing losses. ModGap reports that its gains shrink sharply on imbalanced data such as CIFAR-10. This suggests that forcing uniform spread in a training loss is the wrong tool for skewed concept distributions, so this proposal starts as a post-hoc method.
  • The gap may protect robustness. The robustness paper argues the gap shields representations against noise. Any correction must therefore be checked for robustness loss, broken down by frequency group.

Research Program

Measurement. The first stage is a measurement study on openly trained CLIP-style models whose pretraining captions can be counted. Concept frequency $f_c$ is estimated by counting concept-name mentions in those captions. Concepts come from the OVEN entities in MixBench and the OpenImage-V7 class vocabulary, both of which have natural long tails, plus MSCOCO and Flickr30k objects.

For every concept with enough pairs, I compute $\delta_c$ and three summary quantities:

  • its size,
  • its angle to the global gap,
  • the part of it left over after global centering.

I then ask how much of that leftover variation $\phi(\log f_c)$ explains on held-out concepts. A permutation test that shuffles frequencies across concepts sets the chance level.

Causal test. Because the frequency literature is mostly correlational, the second stage trains small dual encoders on image-caption data. The concepts are resampled to follow Zipf distributions of varying steepness, and a uniform-frequency control is included. The question is whether changing frequency by itself changes $\delta_c$.

Correction. The third stage fits $D$ by least squares on head concepts only. At test time each embedding is corrected with the $\delta$ predicted from its concept's frequency. The concept is identified from the text side, or from the query's predicted label on the image side.

Baselines

Raw CLIP embeddings (do nothing). This is the naive reference point.

GR-CLIP global centering. This is the strongest existing post-hoc competitor, because it is cheap and already highly effective. On MixBench it improves NDCG@10 by up to 26 percentage points over standard CLIP and beats VLM2Vec by about 4 points (Closing the Modality Gap for Mixed Modality Search). It will use the identical calibration split. Beating it means a significant NDCG@10 gain on tail-concept queries with no loss on head queries.

I0T_post batch standardization. On Flickr30k it raises image-to-text retrieval from 69.60% to 73.30% and text-to-image retrieval from 67.10% to 76.30% (I0T). Rerunning it on Flickr30k checks that the correction does not hurt a general benchmark.

Per-concept oracle centering. This subtracts each concept's own measured $\delta_c$. It serves as an upper bound where calibration pairs exist and is undefined where they do not.

Frequency-shuffled ablation. This is the same model with frequencies permuted across concepts. It isolates whether frequency, rather than extra parameters, carries the gain.

For clustering I report V-Measure against ModGap's MSCOCO result of 23.63 (ModGap). ModGap is a training-time method, so this is a reference, not a like-for-like competitor. On the OpenImage-V7 long tail, no reported result was found on Lacuna for any gap method.

All post-hoc methods get the same frozen backbone, the same calibration pairs, and the same held-out concept split.

Why not just…?

Why not just use global centering? It already applies frequency weighting implicitly, as argued above. What it cannot do is let the offset vary by concept. If the measurement stage finds that $\phi(\log f)$ explains no held-out variation beyond the shuffled control, global centering is sufficient. That would be a publishable negative result, and I would abandon the correction.

Why not just learn a linear map from image to text embeddings on the calibration pairs? This is the real threat, since a full linear map can absorb much of the heterogeneity. It is included as a baseline, trained on the same pairs. The proposal claims an advantage only on held-out tail concepts that have zero calibration pairs. If the linear map matches the frequency-conditioned correction there, the frequency hypothesis adds nothing, and I would report that honestly.

Why not just center each concept or cluster separately? Per-concept centering is the oracle above and needs pairs for every concept. Clustering embeddings with k-means and centering each cluster avoids that requirement, but clusters in the tail are dominated by head neighbors. Its tail-group performance is measured directly against the frequency-conditioned correction.

Why not just retrain with ModGap or uniformity losses? ModGap's own authors report weak gains under imbalance. The synthetic training stage tests ModGap at each Zipf steepness to show where it degrades.

Proposed Experiments

Retrieval. The main evaluation is NDCG@10 on MixBench, reported separately by frequency group of OVEN entities, which splits the headline number into head and tail. It is complemented by zero-shot accuracy on long-tail OpenImage-V7 classes.

Robustness. Robustness under Gaussian noise and caption rephrasing on ImageNet follows the protocol of the robustness paper and is also reported per frequency group. This tests whether closing the gap costs rare concepts more robustness than common ones.

Decisive checks. There are two:

  1. The frequency features must explain held-out gap variation well beyond the shuffled control, and do so across at least two backbones.
  2. Frequency-conditioned correction must beat both GR-CLIP and the learned linear map on tail concepts that have no calibration pairs.

If the first check fails, the answer is that the gap is homogeneous with respect to frequency. If the first passes and the second fails, the heterogeneity is real but not exploitable.