Image-text models such as CLIP embed pictures and captions into one nominally shared space. Two failures of this space have been studied separately. The first is the modality gap: image embeddings and text embeddings sit in separate clusters with empty space between them. The second is the bag-of-words failure: the models cannot tell "a red cube and a blue sphere" from "a blue cube and a red sphere." The puzzle is that the text encoder alone, and the image encoder alone, both do encode who-has-which-attribute. Frozen CLIP embeddings separate these bindings with up to 99% linear-probe accuracy, and the information is lost only when the two modalities are matched (CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally; Why is Winoground Hard?). We propose that these two failures share a location. The directions in each embedding that encode word order and binding (call them the structure subspace) are mostly directions the other modality does not share. A large part of the modality gap should therefore sit there, while the directions carrying lexical content (which objects and colors are present) are already close to aligned.
The idea borrows its central trick from semantic-syntactic disentanglement, the line of language work that splits a sentence representation into a "meaning" part and a "form" part. A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations isolates meaning by asking which parts of a representation stay fixed across paraphrases. We run that logic in reverse and across modalities. A caption and its word-swapped counterpart contain exactly the same words, so the difference between their embeddings contains no lexical content and only structure. The same holds for an image and its attribute-swapped counterpart. Collecting such differences defines a structure subspace in each modality with no parser and no generative model. We first measure how the modality gap splits between structure and content. We then learn a low-rank map that aligns only the structure subspaces and leaves the content directions untouched.
Core Claim
The claim has two parts, and either can fail on its own. The diagnostic part is that, in frozen CLIP, the centroid gap projected onto the swap-derived structure subspace is disproportionately large relative to that subspace's dimension. It also predicts that the text structure directions and the image structure directions have low canonical correlation, whereas the content directions correlate well. The method part is that a map acting only within these subspaces recovers binding without the zero-shot classification penalty that the full-vector LABCLIP adapter pays. It should also transfer across swap types better than whole-model hard-negative fine-tuning, which failed to generalize from ColorSwap to Winoground (ColorSwap).
The closest existing work differs in the following ways. LABCLIP (Koishigarina et al.) learns one unconstrained 512×512 matrix on text embeddings. It does not factor structure from content, and it does not measure the gap. Gap-closing methods pull whole-caption embeddings onto whole-image embeddings. Closing the Modality Gap Aligns Group-Wise Semantics does this with explicit alignment plus centroid uniformity, and Its Not a Modality Gap does it with alignment and uniformity terms. Neither is evaluated on any compositionality benchmark. SHE-Net routes video attention with part-of-speech rules taken from a parser. It imposes text structure on vision in one direction only, learns no latent factorization, and never measures gap geometry. The explanations of the gap based on initialization "cones" and contrastive loss dynamics (Mind the Gap; Towards understanding the modality gap in CLIP) motivate a geometric analysis. They treat the gap as a single scalar, and our decomposition refines rather than contradicts them.
We deliberately do not build a dual-bottleneck variational text encoder of the kind used in SEPARATOR (Factorising Meaning and Form), QKVAE (Exploiting Inductive Bias in Transformers), or graph-injected Optimus (Graph-Induced Syntactic-Semantic Spaces). Those papers report that such encoders lose fine-grained lexical detail, suffer posterior collapse, and need large paraphrase corpora. The probing results above suggest the structure information already exists in frozen encoders, so retraining is the wrong first move.
Research Program
Estimating the structure subspaces. Let $f_T$ and $f_V$ be the frozen, normalized CLIP text and image encoders. For word-swap caption pairs $(c, \tilde c)$ we collect differences $f_T(c)-f_T(\tilde c)$. For attribute-swap image pairs $(x, \tilde x)$ we collect $f_V(x)-f_V(\tilde x)$. The image pairs come from the rendered scenes of CLEVR and PUG:SPAR/PUG:SPARE and from the ColorSwap training images. Principal components of these differences give orthonormal bases $U_T, U_V \in \mathbb{R}^{d\times k}$ with $k \ll d$. The complements define content subspaces. Swap pairs cover color binding, other attribute binding, and spatial relations, and we estimate a separate subspace per type as well as a pooled one. This lets us ask whether a single shared structure subspace exists at all.
Decomposing the gap. With text and image centroids $\mu_T$ and $\mu_V$ computed on COCO, we report the per-dimension gap in each subspace, $$g_S = \frac{\lVert U_T^\top(\mu_T-\mu_V)\rVert^2}{k}, \qquad g_C = \frac{\lVert (I-U_TU_T^\top)(\mu_T-\mu_V)\rVert^2}{d-k},$$ alongside the canonical correlations between $U_T^\top f_T(c)$ and $U_V^\top f_V(x)$ on paired data. The diagnostic prediction is $g_S \gg g_C$ together with low structure-subspace correlation. The same measurement is repeated on SigLIP and on the gap-closed models, to see whether whole-vector gap closing shrinks $g_S$ or only $g_C$.
Aligning structure only. We learn a low-rank map $W \in \mathbb{R}^{k\times k}$ that sends image structure coordinates onto text structure coordinates. Scores are computed as $s(x,c)=\langle P_C f_V(x), P_C f_T(c)\rangle + \lambda \langle W U_V^\top f_V(x), U_T^\top f_T(c)\rangle$, where $P_C$ projects onto the content subspace and $\lambda$ weights the structure term. $W$ is trained with a hard-negative contrastive loss over swap pairs. Because the content term is untouched by training, single-object zero-shot classification should be preserved by construction. That is the property we test against LABCLIP. An optional variant also applies a content-only centroid shift in the spirit of ModGap. This tests whether closing the gap in content directions alone recovers the V-Measure clustering gains without harming binding.
Baselines
The naive reference is frozen CLIP ViT-B/32 with no adaptation. On CLEVR binding it scores 58% (Koishigarina et al.), and on ColorSwap it reaches a Group Score of 11.67–12% (ColorSwap). Winoground chance on Group Score is 16.67%, and large models such as FLAVA (14.25%) and VinVL (14.50%) fall below it (Winoground). The file contains no reported Winoground Group Score for CLIP ViT-B/32 itself, so we will measure it.
The strongest direct competitor is LABCLIP, a single 512×512 linear map on frozen text embeddings trained with attribute-swap hard negatives. It reaches 95% on CLEVR (Koishigarina et al.), and the file reports no exact ARO figure for it. It is the right competitor because it is equally cheap, uses the same frozen encoders, and targets the same failure. We will retrain it on exactly our swap data, with the same optimizer budget and hyperparameter search. Beating it means two things. We must match its CLEVR and ColorSwap scores within noise while losing essentially nothing on zero-shot CIFAR-10 and Caltech101, where LABCLIP is reported to pay a penalty. We must also score higher on held-out swap types, especially Winoground and the spatial-relation splits that LABCLIP is reported to fail.
The second competitor is whole-model hard-negative fine-tuning, trained on the same pairs under matched compute. Its reference point is CLIP fine-tuned on 1,400 ColorSwap pairs, which reached 63% ColorSwap Group Score but did not transfer to Winoground (ColorSwap).
The third competitor is gap closing. It includes the post-hoc centroid shift of Mind the Gap and the ModGap objective, which reduced the COCO gap from 0.47 to 0.03 and raised COCO V-Measure from about 13.0 to 23.63 (Closing the Modality Gap). These are evaluated here for the first time on binding benchmarks.
The cross-attention matcher BLIP ITM, at 87% ColorSwap Group Score, serves as a ceiling. It is not a dual encoder, so it is not a like-for-like competitor.
The key ablation gives LABCLIP's full-rank map the same training data but restricts it to rank $k$ without the swap-derived bases. We also try replacing $U_T, U_V$ with random $k$-dimensional subspaces. If these match our method, then the factorization adds nothing beyond capacity control.
Why not just…?
Why not just use LABCLIP's full linear adapter? This is the real threat. A full-rank map trained on swap negatives could learn to act mostly inside the structure directions on its own. We will settle this empirically. We will compute how much of LABCLIP's learned update $A-I$ falls inside $U_T$. We will also compare zero-shot retention and cross-type transfer at matched rank. If LABCLIP's update is already concentrated in the structure subspace and it does not lose zero-shot accuracy under our training data, then the method contribution collapses. What remains is the diagnostic finding, and we would report that honestly.
Why not just fine-tune CLIP on hard-negative swaps? The evidence says this learns the swap distribution it is shown: gains on ColorSwap did not reach Winoground (ColorSwap). It is also reported to be over 100× slower than a linear adapter (Koishigarina et al.). Our leave-one-swap-type-out protocol tests directly whether a shared structure subspace transfers where full fine-tuning does not. If fine-tuning transfers equally well at matched compute, our advantage reduces to cost.
Why not just close the modality gap? Retrieval depends on ranks, so moving an entire embedding cloud does not reorder two captions that share the same words (Closing the Modality Gap). Whole-vector alignment could even reinforce keyword matching. We will measure this rather than assume it by running the gap-closed models through Winoground, ColorSwap, and ARO and reporting $g_S$ versus $g_C$ for each.
Why not just scale? GPT-4V still misses 28% of ColorSwap, and pretrained models at scale sit below chance on Winoground (ColorSwap; Winoground). We also repeat the diagnostic on larger CLIP and SigLIP checkpoints. If the structure-subspace gap shrinks with scale, that would indicate scaling eventually addresses the problem this proposal targets.
Proposed Experiments
The first decisive check is the diagnostic on frozen CLIP and SigLIP using COCO pairs. If $g_S$ is not clearly larger than $g_C$, or if the text and image structure subspaces are already well correlated, the central hypothesis is wrong, and the failure lies in how the score is read out rather than in geometry. The second check is transfer. $W$ is trained on color swaps from ColorSwap and CLEVR only. It is then tested on PUG:SPARE relation splits, ARO relation and order splits, and Winoground, which is never used for training, reporting Group, Text, and Image scores. The third check is retention: zero-shot accuracy on CIFAR-10 and Caltech101 and Recall@K on COCO retrieval, compared against LABCLIP under identical training. The fourth check is gap interaction. We apply a content-only centroid shift and a whole-vector shift and compare COCO V-Measure and binding scores. The prediction is that the content-only shift keeps most of ModGap's clustering gain while leaving binding intact, whereas the whole-vector version does not improve binding. Every result is reported over several seeds and several subspace ranks $k$. The random-subspace ablation is reported beside each, so that any gain can be attributed to the swap-derived subspaces rather than to low rank alone.
References
- CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally
- Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
- A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
- ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation
- Closing the Modality Gap Aligns Group-Wise Semantics
- Its Not a Modality Gap: Characterizing and Addressing the Contrastive Gap
- SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Towards understanding the modality gap in CLIP
- Factorising Meaning and Form for Intent-Preserving Paraphrasing
- Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs
- Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality