Vision-language models such as CLIP embed images and captions into one shared space and score a match by how close the two embeddings are. They are poor at binding. Shown a red car beside a blue suitcase, CLIP cannot reliably prefer "a red car and a blue suitcase" over "a blue car and a red suitcase." On SugarCrepe++, CLIP-style models score roughly 45–55% on such swap probes, while humans score 96–100% (SUGARCREPE++). A separate literature studies the modality gap, the observation that image and text embeddings occupy separate regions of the shared space rather than overlapping (Mind the Gap). It is tempting to blame the binding failure on this gap, and to fix it by closing the gap. We argue that this is the wrong target. For ranking captions against a fixed image, the gap acts mostly as a constant offset, and a constant offset cannot change which caption wins. The more plausible culprit is a training-dynamics effect from work on systematic generalization: when parts of a network that should specialize share a training signal with a holistic, non-compositional target, they couple. That is, they adapt to each other and to the joint statistics of the training data instead of learning reusable pieces (On The Specialization of Neural Modules). CLIP's contrastive objective is exactly such a holistic target, because each whole caption is treated as one indivisible label.
The proposal has two parts. The first is a short argument and test showing that manipulating the gap, whether closing it, widening it, or translating it, leaves swap accuracy essentially unchanged. The second is a phrase-level binding module attached to a frozen CLIP. It is trained only on phrase-level contrasts, and its gradients are explicitly cut off from the whole-caption loss. A matched ablation that allows those gradients through isolates whether segregation is what fixes binding. If the segregated module beats both gap-closing methods and an equally trained hard-negative CLIP on swap probes, it supports the claim that binding is a coupling problem, and it shows how to fix it without retraining the backbone.
Core Claim
Our first claim concerns geometry. Let $s(I,T)=\langle z_I, z_T\rangle$ be the score between image embedding $z_I$ and caption embedding $z_T$. Shifting every caption embedding by the same gap vector $\Delta$ changes each score by $\langle z_I,\Delta\rangle$, an amount that does not depend on which caption $T$ is being scored:
$$ s(I, T+\Delta) - s(I, T') - \big(s(I,T+\Delta)-s(I,T'+\Delta)\big)\;=\; s(I,T)-s(I,T') . $$
The ranking of candidate captions for a fixed image is therefore unchanged. The text-to-text comparison inside SugarCrepe++'s 3-way hit metric is also untouched, because it happens within one modality. Re-normalizing embeddings to unit length after the shift breaks this exactly, but only at second order. This matches the distinction drawn in Closing the Modality Gap Aligns Group-Wise Semantics, which shows that ranking-based retrieval is insensitive to the gap while distance-based clustering is sensitive to it. The prediction is that gap interventions move clustering and noise-robustness metrics but leave swap accuracy within noise.
Our second claim concerns training. Swap failure arises because whatever part of the model represents "red-of-car" receives gradient from a whole-caption target. That target can be satisfied by bag-of-words co-occurrence, so the binding signal is drowned out. The analysis of Jarvis et al. predicts that only strict segregation of the compositional pathway from the holistic signal prevents this coupling. Partial separation, such as parallel adapters fused by soft gating, should not be enough.
Relation to Prior Work
Two works motivate the proposal. SugarCrepe and SugarCrepe++ supply the failure and the evaluation. SugarCrepe also supplies a constraint on design: it showed that NegCLIP's hard-negative gains came from artifacts in older benchmarks and vanished once text-only shortcuts were filtered out (SugarCrepe). Our training negatives therefore must not reward caption plausibility.
The mechanism comes from the specialization analysis of Jarvis et al.. That analysis is proven only for deep linear networks and validated on Compositional MNIST, a digit dataset that requires both per-digit and whole-number labels. We test whether its prediction carries over to a nonlinear, pretrained, cross-modal system. The prediction is that segregated pathways generalize and coupled ones do not.
The modality-gap literature constrains the design. Liang et al. trace the gap to the narrow cone of embeddings that deep networks produce at initialization and to the repulsive effect of the contrastive temperature. They show that shifting the gap tunes zero-shot accuracy, from 0.9013 to 0.9081 on CIFAR-10 (Mind the Gap). Is the Modality Gap a Bug or a Feature? proves that the gap is orthogonal to each modality's own subspace and reports up to 10% better noise robustness from it. We treat that robustness as something the binding module must not destroy. We therefore keep the backbone gap intact and do not try to close it.
The closest methods are the compositional zero-shot learners. These address recognizing unseen attribute-object pairs such as "sliced apple." CAILA inserts attribute, object, and composition adapters into every CLIP layer and fuses them with a learned gate. VP-CMJL learns visual-side class proxies and decouples attribute from object features with separate cross-attention. Both target single attribute-object labels per image. Both also train their decoupled parts jointly with a whole-composition loss, which is the coupling condition Jarvis et al. warn about. Neither tests binding among several objects in one caption. Compositional Feature Alignment enforces orthogonal class and domain heads in a vision-only encoder. Its staged training, which freezes the heads and then fits the encoder, is the closest precedent for our insistence on structural rather than soft separation, but it is unimodal and handles two factors. Our phrase-to-region scorer borrows weakly supervised grounding ideas from MAF.
Research Program
The first stage is a measurement. We take CLIP and apply three gap interventions. The first is mean-difference translation. The second is orthogonal translation following Chowers et al. The third is ModGap fine-tuning: ModGap is the loss from Closing the Modality Gap that pulls true pairs together and spreads concept centroids apart, and it reportedly shrinks the gap from 0.47 to 0.03 on MSCOCO. We then sweep the gap distance and record, at each setting, the SugarCrepe swap and replace accuracy, the SugarCrepe++ 3-way hit, MSCOCO clustering V-measure (how well embeddings group by semantic category), and Gaussian-noise robustness. The prediction is a flat curve for swap accuracy and moving curves for clustering and robustness. A clearly sloped swap curve would falsify our first claim and redirect the project toward gap geometry.
The second stage builds the binding module on a frozen CLIP. A dependency parser splits each training caption from MSCOCO-train into attribute-object and subject-relation-object phrases. Images from the SugarCrepe evaluation splits, which are drawn from MSCOCO validation, are excluded. Each phrase is encoded by a small text head that reads frozen CLIP token features. The phrase embedding then attends over frozen CLIP image patch features to produce a phrase-specific visual vector. The final score is the frozen CLIP score plus the summed phrase-to-region scores.
The binding module is trained with a phrase-level contrastive loss. Negatives are made by swapping attributes between objects that co-occur in the same caption. The swapped phrases are scored against the same image, so text plausibility gives no advantage. Segregation is enforced by a stop-gradient. The whole-caption contrastive loss, used only to calibrate how the two scores are mixed, cannot update the binding module. The module learns only from phrase-level signals.
We also measure coupling directly, in two ways. The first is context leakage: how much the embedding of "red car" changes when the other phrase in the caption changes. The second is the share of the binding module's weight-update norm that comes from each loss term, a nonlinear analogue of the block-norm diagnostics in Jarvis et al.
Baselines
The naive reference point is frozen CLIP scored zero-shot. On SugarCrepe++ swap tasks this class of model is reported at roughly 45–55% 3-way hit (SUGARCREPE++), and we re-run it in our own evaluation code to fix the exact number.
The gap baselines are post-hoc mean translation, orthogonal translation (Is the Modality Gap a Bug or a Feature?), and ModGap fine-tuning (Closing the Modality Gap). For ModGap we found no swap-accuracy result on Lacuna; its reported evidence is clustering, with MSCOCO V-measure rising from about 13.0 to 23.63. ModGap is trained on the same MSCOCO-train images as our module, with the same number of steps.
The strongest training competitor is hard-negative contrastive fine-tuning in the style of NegCLIP. We strengthen it beyond its published form by using exactly our phrase-swap negatives inserted into whole captions, with the same data, the same number of trainable parameters (adapters or a LoRA of matched size), and an equal hyperparameter-search budget. No reported SugarCrepe swap number for NegCLIP is found on Lacuna; the file states only that its earlier gains vanished on SugarCrepe.
We also include CAILA-style parallel adapters with soft gating, trained on the same phrase data. This baseline tests whether modular structure alone, without hard segregation, is enough.
The key ablation uses the identical binding module, data, and budget, but removes the stop-gradient so that the whole-caption loss also trains the module. This is the coupled condition. The difference between the coupled and segregated variants is the cleanest test of the coupling hypothesis.
As an out-of-budget ceiling, GPT-4V is reported at 83.13% on SugarCrepe SWAP and 90.95% overall (SugarCrepe). This is a much larger generative model, so it is not a fair competitor, but it is a useful marker.
We would count the proposal as successful if the segregated module meets three conditions. It must beat the matched hard-negative baseline and the coupled ablation by a margin larger than seed variance on SugarCrepe++ Swap Object and Swap Attribute. It must keep MSCOCO retrieval R@1 within a point of frozen CLIP. It must retain the noise robustness reported for the intact gap.
Why not just…?
Why not just close or translate the modality gap? The ranking argument above says a constant offset cannot change which caption wins for a fixed image. The first stage tests this directly across a sweep of gap distances. If swap accuracy does move with the gap, we report it, because that would be a substantive finding against our thesis.
Why not just fine-tune CLIP with swap hard negatives? This is the real threat. SugarCrepe's evidence against NegCLIP concerns artifact-laden negatives, not well-built swap negatives. We therefore give this baseline our exact negatives and our full budget. If it matches the segregated module, the modular claim fails, and the honest conclusion is that the data mattered, not the segregation. The coupled-versus-segregated ablation makes this outcome readable either way.
Why not just tune LayerNorm? Tuning LayerNorm in Attention shows that adjusting only normalization parameters, as little as 0.004% of the model, handles cross-modal distribution shift and beats LoRA by 4.3% in its "simple" variant. However, those parameters are shared by every computation in the block, which is precisely the coupled condition. We include it, trained on the same swap-negative data, as a cheap point between frozen and fine-tuned CLIP.
Decisive Checks
Three outcomes jointly decide the question. The first is whether swap accuracy stays flat across the gap sweep while clustering and robustness move. The second is whether the segregated module beats its coupled twin, and whether its context-leakage measure is lower; the coupling account predicts that lower leakage tracks higher swap accuracy across seeds. The third is whether attribute-object binding learned on COCO phrases transfers to unseen pairs on C-GQA and MIT-States, compositional zero-shot benchmarks of attribute-object recognition, without training on those datasets. Lacuna reports no numbers for CAILA or VP-CMJL on these benchmarks, so we re-run both under the same protocol.
If the first check fails, the gap matters for binding. If the second fails, segregation does not matter. Either negative result would be informative on its own.
References
- SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterations
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- On The Specialization of Neural Modules
- Closing the Modality Gap Aligns Group-Wise Semantics
- SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
- Is the Modality Gap a Bug or a Feature? A Robustness Perspective
- CAILA: Concept-Aware Intra-Layer Adapters for Compositional Zero-Shot Learning
- Dual-Modal Prototype Joint Learning for Compositional Zero-Shot Learning
- Enhancing Compositional Generalization via Compositional Feature Alignment
- MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning