When Does a 2D Plot Show a Modality Gap That Isn't There?

Vision-language models such as CLIP embed images and captions into one shared vector space. The standard evidence that the two modalities stay apart is a picture. Researchers run a nonlinear projection method, typically t-SNE or UMAP (tools that squash hundreds of dimensions into a 2D scatter plot by keeping each point's nearest neighbors close), and they see two separate islands, one of images and one of text. The same kinds of plots are then used to argue about what happens inside the gap: whether matched concepts form parallel "stripes" or merged "blobs," and whether closing the gap helps or hurts. Neighbor-embedding methods, however, are known to invent clean cluster boundaries where the data only partially overlap. That leaves open how much of the famous picture is geometry and how much is the drawing tool.

Our proposal is to answer this with a ground-truth knob instead of another plot. In a trained CLIP-style space we move one modality along the gap direction by a controlled amount. When that direction is orthogonal to both modalities' spans, this changes cross-modal distances in a known way and leaves within-modality geometry untouched. Sweeping this knob gives a family of embedding spaces whose true gap we know exactly. We then measure what each projection method draws. Our central prediction is a mismatch in shape. The true gap shrinks smoothly, but the drawn separation of the modalities behaves like a switch: it flips from "two islands" to "one cloud" over a narrow band. The band is set by how many of each point's nearest neighbors come from the other modality, not by how large the gap actually is. If this holds, a large body of visual evidence in the modality-gap debate overstates discreteness, and we can say which visual claims survive.

Core Claim and What Is New

The modality gap is real in the full space, and we do not dispute that. It shows up in distances between modality centroids, 0.66 for standard CLIP on MS-COCO in Its Not a Modality Gap and 0.47 in Closing the Modality Gap Aligns Group-Wise Semantics. It also shows up as linear separability, as documented by Mind the Gap. A proposal claiming "the gap is a hallucination" would be refuted by a linear classifier in an afternoon, so that is not our claim.

Our claim is narrower and testable. The visual magnitude of separation, and the visual structure inside and across the gap (stripes versus blobs, cross-modal neighborhoods), are controlled by a single discrete quantity: the fraction of cross-modal edges in the nearest-neighbor graph that t-SNE and UMAP build. Write $\rho(\lambda)$ for the share of $k$-nearest-neighbor edges that connect an image to a text when the text embeddings are shifted by $-\lambda g$. Here $g$ is the component of the mean text-minus-image difference that is orthogonal to both modalities' spans, and $\lambda \in [0,1]$ runs from the original space ($\lambda = 0$) to a gap-free one ($\lambda = 1$). The true gap ratio (centroid distance divided by mean within-modality radius) falls linearly in $1-\lambda$. We predict that the drawn separation tracks a sigmoid in $\rho$. We further predict that the location of the switch moves with the projection's repulsion strength in the way the contrastive account of t-SNE and UMAP implies.

Two lines of work supply the mechanism we test. From t-SNE to UMAP with contrastive learning shows that UMAP is negative sampling with a very large implicit normalization constant. That constant drives strong repulsion between non-neighbors, and their Neg-t-SNE exposes it as a tunable parameter. We reuse that parameter as an experimental dial, but on cross-modal data, which the paper never examined. Assessing and improving reliability of neighbor embedding methods names "overconfidence-inducing" discontinuities, in which overlapping data are forced into clean separate clusters, and gives a label-free perturbation score to detect them. We use that score, but calibrate it against our known ground truth. The original work had no cross-modal ground truth to check against.

Is the Modality Gap a Bug or a Feature? supplies the knob itself. It proves that the contrastive gap vector is orthogonal to both modalities' spans and uses translation along it as a post-hoc fix. We repurpose the same translation as a controlled experimental intervention rather than a fix.

The closest connecting work is AKRMap. It builds supervised 2D maps of cross-modal embeddings that track evaluation scores, and it explicitly does not ask whether modality separation in 2D reflects the full space. It also trades away neighborhood fidelity, which is exactly the property we audit. The theoretical backdrop is that no fixed 2D map can preserve neighborhoods in general: Compressibility Barriers to Neighborhood-Preserving Data Visualization proves this, and the crowding problem of Hinton and Roweis explains why. Distortion is therefore guaranteed, and the useful question is where it lands. Our bet is that it lands at the modality interface.

Research Program

The first component builds the controlled family. We take public CLIP and SigLIP checkpoints and embed MS-COCO images and captions. We estimate each modality's affine span and the orthogonal gap component $g$, and we generate spaces at roughly twenty values of $\lambda$.

Translation takes points off the unit sphere. We therefore run every experiment twice, once without re-normalizing and once with re-normalization. Re-normalizing slightly perturbs within-modality geometry, and we measure that perturbation directly. As a second ground-truth family, we translate along the full mean difference, including its non-orthogonal part. There the knob no longer leaves intra-modal geometry fixed, which tells us how much the clean result depends on Chowers et al.'s orthogonality.

The second component is measurement, taken on each space both in full dimension and after projection. In full dimension we record the gap ratio, linear-probe separability margin, $\rho(\lambda)$, clustering V-measure with modality-merged clusters, and a hubness skew statistic. The skew statistic follows Balance Act, which reports skewness falling from 0.94 to 0.37 on ActivityNet under its normalization. We use it to check whether hub points sit at the interface.

In 2D we record four quantities:

  • a modality silhouette, measuring how cleanly the drawn modalities separate;
  • a cross-modal pair recall, asking whether an image's own caption falls among its 2D neighbors, compared against its rank in full dimension;
  • a stripe-versus-blob index, the ratio of drawn between-modality to within-concept scatter for each concept;
  • the Liu et al. perturbation and singularity scores for each point.

Projections are PCA, t-SNE, UMAP, and Neg-t-SNE at several repulsion settings, each run with ten random seeds. Running many seeds is how we probe the non-uniqueness documented in The Rashomon Effect for Visualizing High-Dimensional Data: we check whether the stripe/blob verdict flips across equally good layouts of the same space.

The third component is a model and a reporting fix. We fit a simple predictive law of drawn separation as a function of $\rho$ and repulsion strength. If the law holds, we test a "gap-factored" display that separates what 2D can show faithfully from what it cannot. The display projects the gap-removed joint cloud, so semantics can be shown faithfully. It reports the full-dimensional gap ratio as a separate measured number rather than letting it be drawn. We then return to the visual claims in the four gap papers and re-check each against full-dimensional metrics.

The fourth component extends the study to spaces trained differently, to test whether the switching law is specific to one checkpoint. We use the three-modality MSR-VTT setting (video, audio, text) of the ModGap paper, and spaces trained with ModGap or with the uniformity terms of Fahim et al. at small scale if public weights are unavailable. This tests whether the stripes-to-blobs transition that ModGap illustrates is mirrored by, and possibly exaggerated by, the projection.

Baselines

The naive reference point is what the literature already does: default UMAP and t-SNE on the joint image-text cloud, the same practice as Mind the Gap and the ModGap paper. It is run at library defaults, plus a grid over neighbor count, on identical embeddings and seeds. It "loses" if its modality silhouette has lower rank correlation with the true gap ratio across $\lambda$ than a competing display does.

PCA is the linear competitor. It cannot create overconfidence discontinuities, and it likely draws the gap faithfully as a leading direction, at the cost of crowding semantics. It gets the same data and no tuning beyond component choice. We count our display as beating it only if cross-modal pair recall and the stripe/blob index are more faithful while gap tracking is no worse.

Per-modality mean-centering before projection is the obvious practitioner fix. It is run identically.

The strongest existing diagnostic is the LOO-map perturbation score of Liu et al., run with the authors' recommended settings on the same 2D maps. The only reported number for it is an out-of-distribution-detection AUROC of 0.75 on CIFAR-10, against 0.698 for kernel PCA. That result is unimodal and not directly comparable. No reported result was found on Lacuna for any method detecting projection-induced modality separation. We therefore define success as follows. Our known-$\lambda$ ground truth must show whether the perturbation score concentrates at the image-text interface, meaning an interface-versus-interior AUROC clearly above chance. It must also show whether a $\rho$-based predictor explains drawn separation better than the perturbation score alone. Here the baseline and the proposal get the same point budget.

The strongest method on the downstream semantic question is ModGap. On MS-COCO it reduces the gap from 0.47 to 0.03 and raises V-measure from about 13.0 to 23.63, and on MSR-VTT it raises three-modality V-measure from 23.3 to 32.1 (Closing the Modality Gap Aligns Group-Wise Semantics). We do not try to beat ModGap's V-measure. We use it to ask whether its visual transition is larger than its full-dimensional change.

AKRMap is included as a cross-modal supervised visualization. It is supervised here with the full-dimensional gap coordinate and trained on the same subset.

The key-ingredient ablation compares translation along the orthogonal gap with translation along the raw mean difference. If the switching law holds only for the orthogonal knob, the result depends on Chowers et al.'s geometry and we will say so.

Why not just…?

Why not just report full-dimensional metrics and skip plots entirely? For the magnitude of the gap this is correct, and it is a real threat to any "plots are wrong" story. That is why our claim is not about gap existence. Plots persist because they are used to argue about structure: stripes, blobs, which concepts neighbor which. That structure has no single scalar summary. The study will explicitly list which published visual claims are fully captured by centroid distance and V-measure; for those, the plot adds nothing. It will also list which claims rely on the drawing alone. If that second list is empty, our recommendation reduces to "skip the plot," which is still a useful result.

Why not just use PCA, which cannot invent discontinuities? PCA is linear and honest about the gap direction. But linear views are known to misrepresent high-dimensional structure in other ways: Li et al. show that low-dimensional linear slices invent barriers that the full space bypasses. PCA will also crowd within-modality semantics. PCA is a full arm of the sweep, so if it tracks both the gap and cross-modal pairs as well as the gap-factored display, the experiment will show that and we will recommend it.

Why not just center each modality before running UMAP? When the gap is exactly the orthogonal translation, centering removes it exactly and is essentially our gap-factored display without the separately reported gap number. The orthogonal-versus-raw-difference ablation and the trained ModGap and uniformity-regularized spaces test whether real gaps are close enough to pure translations for centering to suffice. If they are, centering plus a reported ratio is the practical outcome.

Why not just apply Liu et al.'s perturbation score as is? We do, as the main diagnostic baseline. Its limitation here is calibration. A high score says a region is unreliable, but not whether the drawn gap is too big, too small, or right. Only a space with known $\lambda$ can answer that, and supplying one is the contribution.

Decisive Checks

The main test is the shape of the curve on MS-COCO with CLIP and SigLIP. Plot drawn modality silhouette under UMAP against the true gap ratio across $\lambda$. A near-linear curve refutes our central claim. A sharp transition whose location aligns with a fixed value of $\rho$ across both models, and shifts as Neg-t-SNE's repulsion is varied, confirms it.

The second check is seed stability of the stripe/blob verdict at fixed $\lambda$. If different seeds or repulsion settings of equally good layouts give opposite verdicts on the same space, stripes-versus-blobs pictures cannot serve as evidence in the bug-or-feature debate.

The third check is whether the perturbation score and hub points concentrate at the modality interface. A concentration would show that projection distortion is not spread evenly but lands exactly where cross-modal claims are made.