Making the Modality Gap a Pure Translation: Cross-Modal Ranking Consistency for Contrastive Image–Text Models

Image–text models such as CLIP are trained so that a photo and its caption land near each other in a shared vector space. In practice the two modalities sit in separate regions of that space, divided by a consistent offset known as the modality gap. Two kinds of fix have emerged. Post-hoc fixes subtract each modality's mean vector. Training-time fixes add losses that drag the two clouds together. Neither kind asks the question that decides which fix is enough: once the offset is removed, do images and captions agree on which items are similar to which? If they do, the gap is a pure translation and centering removes it at no cost. If they do not, the gap contains a relational mismatch that no shift can repair. That mismatch would explain why shifting sometimes damages within-modality neighborhoods and sometimes helps.

We propose to measure this relational mismatch directly and to train it away with a teacher-free objective borrowed from unsupervised sentence embedding. Each image and its caption are treated as two "views" of one item. The model is penalized when the image's ranking of the other batch images disagrees with the caption's ranking of the other batch captions. The core claim is falsifiable. Most of the downstream damage attributed to the modality gap, including fragmented semantic clusters, broken embedding arithmetic and failed mixed-modality fusion, comes from relational mismatch rather than from the centroid offset. A model trained for ranking consistency should therefore make cheap mean-centering sufficient, while keeping the option to leave the gap open where it helps robustness.

Where the idea comes from

The unimodal side supplies the mechanism. RankCSE learns sentence embeddings without labels by taking two dropout-perturbed encodings of the same sentence and minimizing the Jensen–Shannon divergence between their similarity-ranking distributions over the rest of the batch. It reports a 4.11% average Spearman gain over SimCSE on BERT-base STS. We reuse that consistency term but change what the two views are. RankCSE's views are two noisy copies of one sentence in one encoder. Ours are an image and its caption in two encoders, compared through each modality's own within-modality similarities. RankCSE's distillation variant needs a teacher; our consistency term does not.

The multimodal side supplies the motivation and the constraints. Several papers establish that the gap is pervasive and partly structural. Mind the Gap shows it exists even at random initialization because each encoder tower funnels outputs into its own narrow cone. Explaining and Mitigating the Modality Gap shows that contrastive training with a decaying temperature freezes the gap, which then shrinks only logarithmically slowly. It's Not a Modality Gap shows that two identical image encoders trained contrastively also develop a gap. Post-hoc work treats the gap as a translation. I0T traces it to modality-specific peak dimensions and removes it by per-modality standardization. GR-CLIP subtracts calibration-set means before fusing documents. Both leave open whether this damages intra-modal ranking structure, and that is precisely what we measure.

Two papers constrain the design. Is the Modality Gap a Bug or a Feature? argues that an orthogonal gap confers robustness to noise, and Mind the Gap reports that a larger gap reduced denigration bias on FairFace. Our method must therefore not force the gap shut. It should make the gap removable on demand. ModGap shows that retrieval depends only on rankings and is gap-insensitive, while clustering depends on distances and is gap-sensitive. That split tells us to evaluate on distance-based tasks, not only retrieval.

Core Claim

Write the gap as two parts: a centroid offset, which centering removes, and a relational mismatch, which it cannot. Our claim is that the relational mismatch carries most of the cost, and that a symmetric cross-modal ranking-consistency loss reduces it without an external text teacher.

Concretely, take a batch of $N$ paired images $x_i$ and captions $y_i$ with normalized embeddings. For item $i$ we form two distributions over the other items $j \neq i$. The first is the image-side distribution $p_i^{I}(j) \propto \exp(\cos(f(x_i), f(x_j))/\tau')$. The second is the caption-side distribution $p_i^{T}(j) \propto \exp(\cos(g(y_i), g(y_j))/\tau')$. Here $f$ and $g$ are the image and text encoders and $\tau'$ is a fixed sharpness temperature. Training minimizes

$$\mathcal{L} = \mathcal{L}_{\text{InfoNCE}} + \lambda \frac{1}{N}\sum_{i=1}^{N} \mathrm{JS}\big(p_i^{I} \,\|\, p_i^{T}\big),$$

where $\lambda$ weights the consistency term and JS is the Jensen–Shannon divergence. Because both distributions are computed within a modality, the term ignores the centroid offset entirely. It only asks the two clouds to have the same shape.

For diagnosis we define relational mismatch $R$ as one minus the average Spearman correlation between the rows of the image–image and caption–caption similarity matrices on held-out pairs. $R$ is unchanged by any per-modality translation. Low $R$ therefore means centering can close the gap losslessly.

Research Program

The work has three parts.

The first is a measurement study on public CLIP checkpoints, before and after I0T-style centering. We compute centroid distance and $R$ on MSCOCO, Flickr30k and CIFAR-10. We then test whether $R$, rather than centroid distance, predicts the residual failures that centering leaves behind: V-Measure clustering on MSCOCO, SIMAT arithmetic, and MixBench fusion.

The second is training. We fine-tune pretrained CLIP on MSCOCO with the objective above, following the fine-tuning regime used by ModGap and the contrastive-gap paper so the numbers are comparable. We compare a symmetric JS term against a stop-gradient variant in which the caption ranking is a fixed target. The asymmetric variant exists because DALR argues that images carry redundant detail that captions omit. Forcing the image geometry to copy the text geometry could erase visual nuance, and the asymmetric variant lets us see whether that happens.

The third is a robustness audit. Each trained model is evaluated with the gap left open and with the gap closed by centering. This tests the claim that low $R$ lets the practitioner choose between robustness and fusion at inference time rather than at training time.

COCO's five captions per image, which DALR identifies as a source of false negatives, are handled by sampling one caption per image per batch. Remaining near-duplicates are left to the soft ranking target, which does not penalize them the way a hard negative would.

Baselines

The naive reference point is the pretrained CLIP checkpoint, plus the same checkpoint fine-tuned on MSCOCO with plain InfoNCE for the same steps. Every gain must be shown against the fine-tuned version, not just the frozen one.

The cheapest real competitor is post-hoc centering, using I0T's standardization and GR-CLIP's calibration-set mean subtraction. I0T reports Flickr30k image-to-text retrieval of 73.30% versus CLIP's 69.60%, and text-to-image retrieval of 76.30% versus 67.10%, with centroid distance falling from 0.7642 to 0.0102. GR-CLIP reports MixBench NDCG@10 gains of up to 26 points over CLIP. We apply centering to every trained model too, so the comparison is "ours plus centering" against "InfoNCE plus centering" on identical data.

The strongest training-time competitor for distance-based tasks is ModGap. Its loss aligns true pairs directly and enforces uniformity on pair centroids. It reports MSCOCO V-Measure improving from about 13.0 to 23.63, and a gap of 0.47 reduced to 0.03, with stable recall at rank 1. We also run three further training-time baselines. The first is the alignment-plus-cross-modal-uniformity loss from It's Not a Modality Gap, which reports Caltech101 zero-shot accuracy rising from 0.42 to 0.49 and an 18% SIMAT gain. The second is temperature scheduling from Explaining and Mitigating the Modality Gap. The third is a re-implementation of AlignCLIP's SBERT-weighted intra-modal separation term, applied in fine-tuning only, because its from-scratch CC12M pre-training is outside our budget.

All methods use the same backbone, data, number of steps and batch size. Each gets the same hyperparameter search budget of a fixed number of runs over its loss weight and temperature, selected on a held-out COCO split.

Two ablations isolate the key ingredient. The first replaces the JS-over-rankings term with a mean-squared error between the image–image and text–text similarity matrices. This tests whether ranking structure, as opposed to raw similarity values, matters. The second applies ModGap's loss plus our term, to check whether the two are complementary.

Beating ModGap means a higher MSCOCO V-Measure than ModGap under our own re-run, with recall at rank 1 no worse than InfoNCE fine-tuning. Beating centering means a lower $R$ and better SIMAT and MixBench scores after centering than InfoNCE plus centering, outside seed variance across three seeds.

Why not just…?

Why not just mean-center the embeddings after training? This is the most serious threat. If CLIP's $R$ is already small, centering captures nearly all the benefit and our method is unnecessary. I0T explicitly leaves open whether standardization harms intra-modal structure. The first measurement study answers that question before any training. If $R$ after centering does not predict residual clustering and fusion failures, we will report that centering suffices and drop the training component.

Why not just distill similarity structure from a strong sentence encoder, as AlignCLIP does with SBERT? A teacher imports its own errors, a limitation the research file notes for both AlignCLIP and RankCSE distillation. It also aligns images to a different model's text geometry rather than to the CLIP text tower's geometry. Teacher distillation might still win on fine-grained text semantics. The AlignCLIP-style baseline under identical fine-tuning settles this. We additionally report STS Spearman for the CLIP text tower under each method, which shows whether teacher-free consistency degrades or improves the text side.

Why not just use ModGap's centroid-uniformity loss, which already fuses the modalities? ModGap closes the gap by pulling pairs together. Pulling true pairs together does not by itself require that the neighbors of an image match the neighbors of its caption. ModGap's own modest gains on imbalanced CIFAR-10 hint at this, since many images share one caption there and pair-level pulling says little about image-to-image structure. The complementary-combination ablation tests whether our term adds anything on top of ModGap. If it does not, the claim fails.

Why not just raise the temperature so the gap never freezes? Temperature scheduling addresses the size of the gap. It does not target relational mismatch. We expect it to lower centroid distance while leaving $R$ largely unchanged, and measuring both quantities for that baseline tests this directly.

Proposed Experiments

The decisive check is a regression across all trained and post-hoc models. If relational mismatch $R$ explains post-centering V-Measure on MSCOCO, SIMAT scores and MixBench NDCG@10 better than centroid distance does, the decomposition is validated. If centroid distance explains them equally well, it is not.

The second check is whether our model plus centering reaches ModGap's V-Measure while retaining the robustness that comes from leaving the gap open. We evaluate under Gaussian noise and caption rephrasing on CIFAR-100 and ImageNet, following the bug-or-feature protocol, comparing our uncentered model against InfoNCE fine-tuning. ModGap's closed gap cannot be reopened in the same way, so this is where the two methods should differ.

The third check guards the cost DALR warns about. ImageNet linear-probe accuracy and fine-grained zero-shot accuracy on Flowers-102 and Stanford Cars must not fall below InfoNCE fine-tuning. If they do, the symmetric loss is flattening visual detail into caption-level structure. In that case the stop-gradient variant becomes the method of record.

Finally, we run MSR-VTT clustering, for which ModGap reports V-Measure 32.1 against CLIP's 23.3. This tests whether ranking consistency transfers beyond static images.