Image-text models such as CLIP, which embed pictures and captions as single vectors and match them by cosine similarity, are known to treat captions as unordered bags of words. They give "horse eating grass" and "grass eating horse" nearly the same score against the same photo (When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?; Winoground). An older line of work in language semantics already had a cure for exactly this failure in text. Relational words are treated not as vectors but as operators: a verb or preposition becomes a matrix or tensor that acts on its subject and object, so swapping the arguments changes the result. We propose bringing this idea into image-text matching. A relation word in the caption would generate an asymmetric bilinear form, a score of the form $a^\top W_r b$, which is applied to the image regions that the caption's subject and object attend to.
The less obvious part, and the reason to do this now, is geometric. Image embeddings in these models sit in a narrow cone, displaced from text embeddings by a large shared offset, which is the "modality gap" (Mind the Gap). Cosine retrieval is provably insensitive to that offset (Closing the Modality Gap Aligns Group-Wise Semantics), which explains why closing the gap has not been expected to help compositional matching. A bilinear operator, however, is not translation-invariant. Write each region vector as $a = a_0 + g$, where $g$ is the shared cone or gap direction and $a_0$ is the content-specific part. Then $a^\top W b$ contains the constant $g^\top W g$ and the one-argument terms $a_0^\top W g$ and $g^\top W b_0$. These swamp the only order-sensitive part, the antisymmetric interaction $a_0^\top (W - W^\top) b_0$. Our central claim is that the gap is harmless for vector matching but becomes a bottleneck once relations are modeled as operators. Operator-based composition and gap removal should therefore help mainly together, not separately.
Core Claim
Consider two choices: score relations with a low-rank, type-driven operator, and close the modality gap. We predict a positive interaction between them on relation and order benchmarks. The operator should add most when the gap is closed. Closing the gap should add little or nothing to a cosine-scored model, yet add substantially to an operator-scored one. If no such interaction appears, the geometric story is wrong even if the operator head helps on its own, and we will report it that way.
Research Program
Three pieces of prior work supply the method. Adjectives as matrices acting on noun vectors (Nouns are Vectors, Adjectives are Matrices) and verbs as order-3 tensors contracted with subject and object (Learning Type-Driven Tensor-Based Meaning Representations) give us the type-driven form. The parameter explosion of one full tensor per word is avoided with a single shared core tensor $\mathcal{T}$, following the Syntactic RAND-WALK construction of Understanding Composition of Word Embeddings via Tensor Decomposition. That core is factorized at low rank, which Low-Rank Tensors for Verbs in Compositional Distributional Semantics found matches full tensors. Concretely, the relation matrix is $W_r = \mathcal{T} \times_3 v_r$, where $v_r$ is the text encoder's contextual vector for the relation phrase and $\mathcal{T} \in \mathbb{R}^{d \times d \times d_r}$ has CP rank $k$ of roughly 64. This keeps the added parameters near $k(2d + d_r)$ rather than $d^3$.
On the image side, we do not use box supervision. The caption's subject vector $v_s$ and object vector $v_o$ each attend over the CLIP vision transformer's patch tokens, and the attention produces pooled region vectors $a$ and $b$. The final match score is
$$s(I, T) = \cos(z_I, z_T) + \lambda \, \tilde a^\top W_r \tilde b ,$$
where $z_I$ and $z_T$ are the usual global embeddings, $\lambda$ is a learned scale, and $\tilde a$ and $\tilde b$ are the region vectors after the gap-handling step under test. Captions are parsed into subject-relation-object triples with an off-the-shelf dependency parser. When no triple is found, the operator term is zero and the model reduces to CLIP. Attributes are handled the same way, with an adjective-generated matrix acting on one region, which is the matrix-noun case.
Gap handling comes in three strengths so that the geometric claim can be separated from engineering. The first is none. The second is per-modality mean-centering of patch and region vectors, a parameter-free way to subtract $g$. The third is the full ModGap objective, which aligns true pairs by Euclidean distance and enforces uniformity over pair centroids (Closing the Modality Gap Aligns Group-Wise Semantics). For each model we also measure the quantities the argument depends on. These are the centroid gap distance, the share of operator-score variance explained by the constant and one-argument terms, and the antisymmetric norm $\|W_r - W_r^\top\|$ relative to $\|W_r\|$. The interaction claim predicts that the one-argument share falls sharply once the gap is removed.
All variants are fine-tuned from the same CLIP checkpoint on COCO captions, with the same parsed hard negatives (subject-object swaps, attribute swaps, verb substitutions) used by NegCLIP and CE-CLIP. We deliberately avoid training on Visual Genome triples, because ARO's VG-Relation is derived from Visual Genome (Visual Genome).
Baselines
The naive reference point is zero-shot CLIP, which When and Why Vision-Language Models Behave like Bags-Of-Words reports at 63% on VG-Relation and 46% on COCO-Order. It shows how much any fine-tuning buys.
The strongest existing competitor is hard-negative fine-tuning in the plain vector space. NegCLIP is reported in the same paper at 81% on VG-Relation and 86% on COCO-Order, with ImageNet zero-shot preserved at 75% versus 72%. CE-CLIP adds an intra-modal contrastive loss and an adaptive cross-modal ranking loss, and reports a 23.7% relation gain and a 13.5% attribution gain over base CLIP on ARO (Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives). Its Winoground score is not reported in the file. These are the strongest baselines because they attack the same failure with the same data and no new structure. If they already capture everything, the operator is unnecessary. We re-train both on our COCO split, with our parsed negatives and an equal GPU-hour budget, using the same hyperparameter search budget as our own variants. We report published numbers only as sanity checks on the reimplementation.
The ModGap-only baseline applies gap closing to a cosine-scored NegCLIP. It reported a gap reduction from 0.47 to 0.03 and MSCOCO clustering V-Measure rising from about 13.0 to 23.63, with stable retrieval.
Two ablations isolate the key ingredient. The first is a parameter-matched head of the same form whose relation matrix is forced symmetric, $W_r = W_r^\top$. It cannot express argument order, so it should collapse on subject-object swaps. The second is a parameter-matched MLP that receives the same inputs $(a, b, v_r)$ and is trained identically. It tests whether multilinearity, rather than parse-guided region pooling, carries the gain.
Beating the field means three things. On VG-Relation and Winoground Group Score, the operator-plus-gap-closing model should exceed reimplemented NegCLIP and CE-CLIP by more than the seed-to-seed spread across three seeds. The cosine-versus-operator by gap-versus-no-gap interaction should be positive with a bootstrap confidence interval excluding zero. COCO retrieval Recall@1 and ImageNet zero-shot accuracy should stay within one point of NegCLIP.
Why not just…?
Why not just mine harder negatives, as NegCLIP and CE-CLIP do? This is the main threat, because those methods already deliver large ARO gains. Strict minimal pairs remain unsolved, however. On Winoground, VinVL scores 14.50% and FLAVA 14.25% Group Score, against 16.67% chance and 85.5% for humans (Winoground). CE-CLIP also notes that its signal is text-side only and that scale does not fix it. Our design gives the operator models exactly the same negatives. Any gain is therefore attributable to the scoring form, and if the gains are equal we will conclude that negatives suffice.
Why not just close the gap with ModGap, or simply center the embeddings? The ModGap paper itself proves that rank-based matching is gap-insensitive, so for a cosine model we expect gap closing to leave relation accuracy unchanged. That expectation is one cell of our 2×2 design. Mean-centering is a real threat to the "full ModGap is needed" part of the claim. If centering gives the same interaction, we will report it, since it would be a cheaper and equally interesting result. It would still mean the gap matters only for operator scoring.
Why not just add a cross-attention or MLP head over the same region and relation features? A flexible head could learn order sensitivity without tensor structure. The parameter-matched MLP ablation answers this directly. If the MLP matches the bilinear operator, the tensor-semantics contribution reduces to "parse-guided pooling helps", and the geometric decomposition argument loses force, because an MLP has no clean one-argument-term analysis.
Why not just scale the model? CE-CLIP reports that ViT-G/14 trained on billions of pairs is worse on compositional tasks than its much smaller model. Mind the Gap traces the gap to the cone effect and to contrastive dynamics rather than to model size. Scaling is therefore not expected to remove either problem, and we do not test it.
Proposed Experiments
The decisive experiment is the full grid: {cosine, symmetric operator, MLP head, asymmetric operator} crossed with {no gap handling, centering, ModGap}, all trained on identical COCO data with matched negatives and compute. We evaluate on ARO (VG-Relation, VG-Attribution, COCO-Order, Flickr30k-Order), on Winoground Text, Image and Group scores, and on SugarCrepe, VALSE and VL-CheckList. Retrieval is measured as COCO Recall@1, clustering as MSCOCO V-Measure, and transfer as ImageNet zero-shot accuracy.
Two confounds get explicit treatment. First, shuffled-order foils can break the parser. We therefore parse every candidate caption with the same procedure, report the parse-failure rate per benchmark, and also score the order subsets with parse failures counted as cosine-only, so that the operator gains no advantage from parser errors. Second, Winoground captions often lack a clean triple, so we report Winoground separately on the parseable subset and on the full set.
The mechanistic check tests whether the one-argument share of operator-score variance falls with gap removal, and whether that fall predicts the per-model gain on VG-Relation. If the operator wins without the predicted interaction, we keep the method and drop the geometric explanation. If neither the operator nor the interaction appears beyond matched hard negatives, the result is a clear negative: type-driven composition does not survive transfer into contrastive image-text spaces.
References
- When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Closing the Modality Gap Aligns Group-Wise Semantics
- Nouns are Vectors, Adjectives are Matrices: Representing Adjective-Noun Constructions in Semantic Space
- Learning Type-Driven Tensor-Based Meaning Representations
- Understanding Composition of Word Embeddings via Tensor Decomposition
- Low-Rank Tensors for Verbs in Compositional Distributional Semantics
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding