Closing the Modality Gap Without Pairs

Image and text encoders trained separately, or even jointly, end up placing their embeddings in different regions of the shared space. Captions sit in one narrow cone and images in another. Prior work has shown this "modality gap" damages tasks that rely on distances, such as grouping concepts across modalities. Every method that closes the gap, however, pulls each image toward its own caption, so it needs paired data. Multi-view clustering faces a related problem: data from uncoordinated sensors or databases arrives shuffled, and nobody knows which record in one view matches which record in the other. We ask whether the gap can be closed when the pairs are unknown.

We propose a two-stage method. Stage one matches groups by their shape: cluster each modality, then compare how the cluster centroids are arranged relative to each other. Stage two matches statistics within each matched group pair, aligning means, variances and higher-order moments so the two modalities overlap locally rather than only on average. The matched groups prevent the class mix-up that global distribution matching causes. The local moment matching removes the offset between modalities that relational matching alone cannot see. Both stages run in linear or near-linear time in the number of samples. If the method works, data collected without pairs could still yield a shared space whose distances carry meaning.

Core Claim

The modality gap and the correspondence problem have different causes, so we solve them with different tools. The gap is largely a low-order statistical offset. Mind the Gap traces it to the narrow cones each encoder produces from initialization. Ex-MCR finds the within-model gap is linear. LayerNorm tuning treats the shift to a new modality as something fixed by recentering and rescaling. An offset like this can be measured and removed from unpaired samples. Correspondence, by contrast, is relational: which concept in one modality plays the same structural role as a concept in the other.

Our claim is that cluster-conditional moment matching, guided by a relational coupling between clusters, closes the gap with unpaired data. Concretely, we expect it to beat three alternatives on cross-modal grouping (V-Measure): global moment matching alone, relational matching followed by a rigid rotation, and alignment-free centroid clustering. We also expect it to recover a meaningful share of the grouping quality that paired gap-closing methods reach.

Relation to Prior Work

Three paired methods motivate the problem.

  • Mitigate the Gap (AlignCLIP) shares encoder layers and uses language similarities to spread images apart.
  • ModGap pulls true pairs together in Euclidean distance while spreading pair centroids apart. It shows clustering is gap-sensitive while retrieval is not.
  • CaReFlow transports one modality's distribution onto another along straight flows, guided by labels.

All three need pairs or labels to decide which points to pull together. We replace the pairs with a coupling inferred from structure.

Three relational-matching methods supply our technique.

  • Gromov-Wasserstein word alignment matches two languages' word-similarity matrices without a dictionary and then fits a rotation. Gromov-Wasserstein (GW) matching compares the internal distance structure of two point sets rather than their coordinates.
  • Graph Optimal Transport combines node-level and relational matching, but within paired image-caption examples.
  • OPTION uses a GW-style structural loss for unaligned multi-view clustering. That loss costs $O(V^2 B^2)$ for $V$ views and batch size $B$. OPTION also does not examine gap geometry.

We reuse GW matching, but only between a few hundred cluster centroids rather than all samples. This follows the anchor idea of Li et al..

On the distribution side, Central Moment Discrepancy (CMD) matches central moments order by order at linear cost, but has only been applied to unimodal domain adaptation. It also needs bounded activations. Our setting satisfies this for free: embeddings normalized to the unit sphere have every coordinate in $[-1,1]$. ReViSE applied global kernel MMD between visual and attribute embeddings, but with paired training. Direction 2's local-MMD papers show why global matching risks mapping class A onto class B; our cluster coupling is the unsupervised replacement for their class labels.

Two papers constrain the design.

  • AF-UMC argues that explicit cross-view alignment is error-prone and that pulling samples toward other views' cluster centroids suffices. That is a direct challenge, and we test it head on.
  • Mind the Gap shows an intermediate gap can help zero-shot accuracy and reduce demographic harms. We therefore treat the residual gap as a controllable quantity, not something to drive to zero.

Research Program

Inputs are two unpaired collections, $X$ (for example, images) and $Y$ (for example, sentences), each embedded by a frozen encoder trained on its own modality. We learn maps $f_a, f_b$ into a common $d$-dimensional sphere. Following Ex-MCR's finding that nonlinear layers distort leaf geometry, each map is linear, with an optional small MLP as an ablation. Training has three steps.

Step 1: cluster each side. We cluster each modality into $K$ groups with centroids $\{c^a_k\}$ and $\{c^b_l\}$.

Step 2: couple the clusters. We solve entropic GW between the centroid cosine-distance matrices with Sinkhorn iterations. This yields a soft coupling $T \in \mathbb{R}^{K \times K}$ at cost $O(K^3)$, independent of $N$. To break the symmetric-subgraph ambiguity that Wen et al.-style matching suffers, we run several restarts. We keep the coupling with the lowest GW cost after the cluster statistics have been aligned, which tests the restarts against the second-moment structure rather than geometry alone.

Step 3: align distributions within matched cluster pairs. We train the maps to minimize

$$ \mathcal{L} = \sum_{k,l} T_{kl}\,\mathrm{CMD}_K\big(f_a(X_k), f_b(Y_l)\big) + \lambda\,\mathcal{L}_{\text{unif}}, $$

where $X_k$ and $Y_l$ are the members of clusters $k$ and $l$. $\mathrm{CMD}_K$ is the sum of differences of the first $K$ central moments, with $K$ around 5 following Zellinger et al. $\mathcal{L}_{\text{unif}}$ is ModGap-style centroid uniformity, which prevents everything from collapsing to one point. We then refit clusters in the shared space and update $T$, alternating for a few rounds.

A residual-gap knob, which down-weights the first-moment term, lets us trace the trade-off between gap distance and downstream behavior.

Baselines

The naive reference is the frozen encoders with only per-modality centering and normalization, plus naive mean shifting. AlignCLIP reports that mean shifting distorts intra-modal structure. The paired upper reference is ModGap trained on true MSCOCO pairs. It reports V-Measure rising from about 13.0 for CLIP to 23.63 and gap distance falling from 0.47 to 0.03 (ModGap). We do not expect to beat it; the question is what fraction of the 13.0-to-23.63 improvement we recover without pairs. We will report this reference alongside our runs using the same frozen encoders, so the comparison isolates the loss of pairing.

The strongest unpaired competitor is a direct transfer of Alvarez-Melis & Jaakkola to these embeddings: whitening, then sample-level GW on a large subsample, then an orthogonal rotation. In its home domain it reaches about 81–82% Precision@1 on MUSE EN-ES and EN-FR. No reported result was found on Lacuna for it on image-text embeddings.

For the multi-view clustering track, the strongest existing methods are OPTION and AF-UMC.

  • OPTION reports 46.3% accuracy on BDGP with 60% of samples shuffled, and 74.7% on CUB with 70% missing views (OPTION).
  • AF-UMC reports ACC 0.1625 on the 101k-sample YoutubeFace, where competitors ran out of memory, and a 19.57% ACC gain over the runner-up on Hdigit (AF-UMC).

All methods get the same frozen features or raw views, the same embedding dimension, and the same hyperparameter search budget of a fixed number of trials on a held-out split with no labels. They also share one GPU-hour budget per dataset. Beating a baseline means a V-Measure or clustering-accuracy gain larger than the standard deviation across five seeds.

Three ablations isolate the key ingredients:

  • global CMD with no cluster coupling, testing whether matching groups matters;
  • the cluster coupling with mean-only matching, testing whether higher moments matter;
  • the cluster coupling with sample-level attraction to the soft pairs implied by $T$ in place of CMD, testing whether distribution matching beats pseudo-pair pulling.

Why not just…?

Why not just whiten both sides, run GW, and fit a rotation, as in cross-lingual word alignment? This is the real threat. Whitening already removes first- and second-moment offsets, so if it works, our moment matching adds little. Two things could make it fall short. First, a rotation cannot remove anisotropic, cluster-specific offsets. Second, sample-level GW is $O(N^2)$ in memory and is prone to flipped sub-clusters when geometry is symmetric. The direct comparison on MSCOCO and MSR-VTT, at matched compute, settles this. If whitening plus GW plus rotation comes within seed noise of our method, we will report that the gap is effectively a second-order offset solvable by classical tools. That would be a useful negative result, and we would recast the paper as that finding.

Why not just match global distributions with CMD or MMD and skip correspondence? Matching marginals is blind to which cluster lands where, and the Direction 2 local-MMD literature exists precisely because global matching mixes classes. The global-CMD ablation measures this directly. We expect a small gap distance but low V-Measure, since cross-modal cluster agreement should suffer when clusters are crossed.

Why not just use AF-UMC's alignment-free centroid pull? AF-UMC argues alignment is error-prone. Its centroid pull does implicitly assume cluster k in one view corresponds to cluster k in another, which is exactly the correspondence our coupling estimates explicitly. We run AF-UMC on the same features and views. On the multi-view benchmarks where it is strong, we must at least match it, and we report the comparison per dataset rather than averaged.

Why not just build pseudo-pairs through an existing aligned model, as Ex-MCR does? That works when a paired-trained bridge exists, but it imports the bridge's own gap and biases and is not available for sensor or scientific modalities. We include it as an informative reference with a separate label. It is not a competitor in the strictly unpaired setting.

Proposed Experiments

The main testbed is MSCOCO and MSR-VTT with captions randomly permuted relative to images (and audio), so no pairs are available at training time. Metrics are:

  • gap distance, defined as in ModGap;
  • cross-modal clustering V-Measure;
  • correspondence recovery, measured post hoc as the fraction of images whose nearest mapped caption is their true one;
  • retrieval R@1, which ModGap predicts should be gap-insensitive.

The second track runs BDGP, Hdigit, CUB and YoutubeFace under the shuffle and missing-view rates used by OPTION and AF-UMC, reporting accuracy against their published numbers above.

Scaling check. Wall-clock time and memory against sample-level GW as $N$ grows to YoutubeFace size. The cluster coupling should keep cost flat in $N$.

Ambiguity check. Synthetic views with deliberately symmetric cluster layouts, measuring how often each method flips sub-clusters.

Gap-control check. Sweeping the residual-gap knob and plotting V-Measure against CIFAR-10 zero-shot accuracy. This tests in the unpaired regime whether Mind the Gap's finding holds: a nonzero gap helps classification while hurting grouping. That paper reports CIFAR-10 moving from 0.9013 to 0.9081 under manual gap tuning.

Decisive outcomes.

  • If the whitening-plus-GW baseline matches us, the gap was second-order.
  • If global CMD matches us, correspondence was unnecessary.
  • If only the full method recovers a substantial fraction of ModGap's paired V-Measure gain, the claim stands.