Models like CLIP place images and captions in one shared vector space, but the two kinds of data never fully mix. Image embeddings sit in one region and text embeddings in another, separated by what the literature calls the modality gap. Two cheap fixes already exist. One subtracts each modality's average vector. The other slides one modality toward the other along a single direction. Both are rigid moves: they shift or rotate a whole cloud of points without changing its shape.
This note asks whether the gap is fully rigid. If it is not, then after centering and rotation the image and text clouds still differ in shape. They would have different spreads along different directions, different cluster tightness, and different local density. We expect that leftover non-rigid mismatch to explain much of what still separates frozen CLIP from a model retrained to close the gap. We propose to measure that residual directly. We then remove it after training with a small learned map that matches the two distributions without tearing the neighborhoods inside each modality. Neither ingredient needs paired data.
Core Claim
Prior work treats the gap as a vector to subtract. Mind the Gap explains why the gap appears: networks compress their outputs into narrow cones at initialization, and the contrastive loss then holds the two cones apart. I0T shows that the gap is carried by a few dimensions with modality-specific peak activations. Subtracting per-modality means cuts the distance between the two centroids from 0.7642 to 0.0102 and raises Flickr30k text-to-image retrieval from 67.10% to 76.30%. Is the Modality Gap a Bug or a Feature? proves that the gap vector is orthogonal to the span of the data. It closes the gap by translating along that orthogonal component, which leaves pairwise distances untouched and improves noise robustness by up to 10%. All three operations are rigid.
The only method in the file that gets large gains on distance-based tasks retrains the encoders. Closing the Modality Gap Aligns Group-Wise Semantics (ModGap) replaces the contrastive loss with a direct pull on true pairs plus a term that spreads class centroids apart. It raises MS COCO clustering V-Measure from about 13.0 to 23.63 and MSR-VTT tri-modal V-Measure from 23.3 to 32.1.
Our claim has two parts, each testable.
- After the best rigid alignment, a statistically detectable shape mismatch remains between image and text embeddings. The measure is a two-sample test using maximum mean discrepancy, defined below, computed within semantic groups.
- Removing that mismatch with a neighborhood-preserving map on frozen embeddings recovers a substantial part of ModGap's clustering gain. It does so without retraining and without the retrieval and robustness losses that unconstrained alignment causes.
If the first part fails, the gap is effectively rigid. That is a clean and useful negative result: training-free centering would then be all a frozen model needs.
Research Program
The first stage is diagnostic. We take frozen CLIP embeddings on MS COCO, Flickr30k and MSR-VTT and apply a ladder of rigid corrections:
- mean centering;
- orthogonal-component translation;
- centering followed by an orthogonal Procrustes rotation fitted on paired captions, which is the best rigid map available when pairs exist.
After each step we measure the remaining mismatch with maximum mean discrepancy (MMD). MMD is a distance between two sample sets: it compares their average similarity under a kernel, and it is zero only when the distributions match. We use a multi-kernel version with several bandwidths so that mismatch at different scales is captured. We compute it both globally and within COCO object-category groups, and we calibrate it against a permutation-test null.
We also decompose the residual into three parts: the part inside the data span, the part orthogonal to it, and the second-order part (covariance mismatch). This tells us where the non-rigid structure lives.
The second stage is the method. We learn a map $g$ applied to text embeddings $t$:
$$g(t) = R\,(t-\mu_t) + \mu_v + h(t),$$
where $\mu_t$ and $\mu_v$ are the text and image means, $R$ is an orthogonal matrix, and $h$ is a small residual network initialized at zero. The map is trained to minimize
$$\mathcal{L} = \mathrm{MMD}^2_{\text{multi-kernel}}\big(g(T), V\big) + \lambda\,\mathrm{JS}\big(P_k(T)\,\|\,P_k(g(T))\big).$$
Here $T$ and $V$ are batches of text and image embeddings. $P_k(\cdot)$ is the multi-hop neighborhood distribution of a $k$-nearest-neighbor similarity graph, following the structure regularizer of STRUCTURE. $\mathrm{JS}$ is the Jensen-Shannon divergence. The first term moves the text distribution onto the image distribution. The second term allows only moves that keep each caption's neighbors its neighbors.
We also test a class-conditional variant of the MMD term, using pseudo-groups formed by clustering the image side. It is motivated by the local, subdomain-level alignment idea surveyed in MEDA. Global matching is known to pull distinct classes together. Preserving Manifold Geometry in Latent Spaces shows the related failure in which unconstrained alignment collapses continuous structure into clusters.
Neither loss term uses pairs. The method can therefore be fit on unpaired image and caption corpora, which Procrustes cannot be.
The roles of prior work are as follows:
- Supplying the method. STRUCTURE contributes the neighborhood regularizer. The cross-modal MMD idea comes from ReViSE, which aligned CNN features with attribute vectors but had no geometric constraint.
- Closest in spirit. MPS-Tuning preserves geometry through Gram-matrix matching while fine-tuning the vision encoder. It never touches the gap or the cross-modal distribution.
- Constraining the design. The orthogonal-gap paper warns that careless closure can cost robustness, so robustness is a required evaluation rather than an afterthought. Ex-MCR found that the within-space gap is mostly linear. That is why the map is mostly rigid, with only a small residual network $h$.
Baselines
Naive reference. Raw frozen CLIP embeddings. On COCO clustering, ModGap reports V-Measure of about 13.0 for standard CLIP. Matching that order with our reimplementation is a sanity check.
Training-free standardization. I0T_post from I0T, using means computed on the training split and stored as fixed statistics. Its reported Flickr30k numbers are I2T 73.30% and T2I 76.30%. We found no reported clustering V-Measure for I0T in the file, so we will measure it.
Orthogonal translation. The method from Is the Modality Gap a Bug or a Feature?. It is the strongest training-free competitor on robustness, since it reports up to 10% gains under Gaussian noise.
Rigid oracle. Centering plus a closed-form orthogonal Procrustes rotation fitted on COCO training pairs. This is the decisive ablation. It isolates whether anything beyond a rigid map is needed. It is also arguably stronger than our method in one respect, because it uses pair supervision that we do not.
Unconstrained MMD. Our map with $\lambda=0$, in the spirit of ReViSE. This isolates the neighborhood term.
Retrained ceiling. ModGap with its reported COCO V-Measure of 23.63 and MSR-VTT V-Measure of 32.1. We will retrain it only if compute allows; otherwise we compare to the reported numbers using the same backbone.
Learned projector without regularization. For the low-data regime, an unregularized linear-plus-MLP projector, as in Ex-MCR.
All post-hoc methods use the same frozen backbone and the same COCO training embeddings. They get the same hyperparameter search budget on a held-out validation split, roughly 20 configurations each, including $k$ and $\lambda$ for ours. Each method is reported over 5 seeds.
Beating the field means three things at once:
- exceeding the rigid oracle's COCO V-Measure by more than seed noise;
- keeping retrieval R@1 within one point of
I0T_post; - matching orthogonal translation on noise robustness.
Recovering at least a third of the gap between CLIP's ~13.0 and ModGap's 23.63, without retraining, would count as a meaningful result.
Why not just…?
Why not just subtract the per-modality mean? Because it fixes the first moment only. A near-zero centroid distance, like I0T's 0.0102, can coexist with very different covariance and cluster shapes. Our diagnostic stage measures exactly that residual. If the within-group MMD after centering falls inside the permutation null, this alternative wins and we will say so.
Why not just rotate after centering, using paired data (Procrustes)? This is the real threat. A rotation preserves every distance inside each modality, so no regularizer is needed, and with COCO pairs it is solvable in closed form. Our claim depends on the residual being non-rigid. The rigid oracle is therefore the main ablation, and our method must beat it on clustering to justify the residual network. We also test an unpaired regime, where captions and images come from disjoint COCO subsets. In that regime Procrustes is unavailable but the MMD and neighborhood terms still apply.
Why not just retrain with ModGap? Retraining gives the largest reported clustering gains. However, it requires access to and compute for contrastive pretraining, and it produces a new model, not a fix for deployed embeddings. ModGap also reports only modest gains on CIFAR-10 when classes are imbalanced. We treat it as the ceiling, not the competitor. The question we answer is how much of its gain is reachable post hoc.
Why not just align distributions with plain MMD? Prior evidence says global matching over-warps. STRUCTURE reports large gains from neighborhood regularization over unregularized bridges, and MEDA documents class mixing under global matching. The $\lambda=0$ ablation tests whether this holds for CLIP's gap specifically. We expect it to show higher V-Measure but lower retrieval and lower mutual-neighbor preservation. If it does not, the neighborhood term is unnecessary.
Proposed Experiments
Experiment 1: the diagnostic. We report within-group multi-kernel MMD with permutation p-values after each rigid correction, on COCO, Flickr30k and MSR-VTT. The residual is decomposed into within-span, orthogonal, and covariance components.
Experiment 2: clustering. Following ModGap's protocol, we run k-means on the pooled image and text embeddings and report V-Measure. This uses COCO categories and the MSR-VTT tri-modal setting. For MSR-VTT, both audio and text are mapped onto the video side.
Experiment 3: retrieval. Flickr30k and COCO I2T and T2I R@1. Retrieval is rank-based and insensitive to the gap, so it should not drop. A drop signals warping.
Experiment 4: geometry preservation. The mutual k-nearest-neighbor overlap between text neighborhoods before and after the map, using STRUCTURE's metric.
Experiment 5: robustness. The orthogonal-gap paper's Gaussian-noise zero-shot protocol on ImageNet and CIFAR-10/100, plus the FairFace denigration measure from Mind the Gap. Mind the Gap reports that enlarging the gap reduced harm, for example from 15.7% to 13.7% for one group. A gap-closing method must therefore show it does not reverse this. We report FairFace results as a constraint, not a target.
Experiment 6: data efficiency. We fit the map on 1,000, 10,000 and 80,000 unpaired samples. These sizes mirror STRUCTURE's low-data regime.
Decisive check. Compare the rigid oracle and our method on COCO V-Measure across seeds, with the diagnostic residual alongside. A clear win, together with a significant post-rigid residual that our map removes, supports the claim that the gap is not rigid. A tie, with a residual already at the null level, settles the question the other way and recommends centering plus rotation as the complete post-hoc fix.
References
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- Is the Modality Gap a Bug or a Feature? A Robustness Perspective
- Closing the Modality Gap Aligns Group-Wise Semantics
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You
- Visual Domain Adaptation with Manifold Embedded Distribution Alignment
- Supervised Domain Adaptation Based on Marginal and Conditional Distributions Alignment
- Learning Robust Visual-Semantic Embeddings
- Preserve and Sculpt: Manifold-Aligned Fine-tuning of Vision-Language Models for Few-Shot Learning
- Extending Multi-modal Contrastive Representations