Where CLIP Loses the Idiom: Localizing and Transporting Non-Compositional Meaning Across the Modality Gap

Ask a vision-language model like CLIP for an image of "break the ice" and it returns frozen water, not two strangers laughing. The failure is severe. On the XMPIE benchmark, the 18-billion-parameter EVA-CLIP ranks the literal image first 90% of the time and the correct figurative image first only 6% of the time (A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding). The systems that do well on idiom-to-image tasks go around the embedding space. They have a large language model rewrite the idiom into plain words, or caption every image and reason entirely in text. None of them asks where, inside the model, the figurative meaning goes missing.

This note proposes a way to answer that question and a cheap fix that follows from the answer. For text-only models, a training-free "synergy" calculation isolates the part of a phrase's embedding that its words do not explain. It works like a discrete derivative: take the whole-phrase embedding and subtract the embeddings of its parts, correcting for overlaps. For idioms, the leftover vector points toward the figurative meaning (A Compositional Calculus for Semantic Synergy in Language Model Embeddings). We compute this residual inside CLIP-style text encoders and ask whether it lives in directions that image embeddings can "see" or in directions only text occupies. Our central hypothesis is that the literal bias comes from a specific place. The text encoder partly computes the idiom's meaning, but places it in a text-only subspace that image similarity cannot use. If that is right, moving just that residual into the shared subspace should shift retrieval toward figurative images in a single forward pass, with no language-model paraphraser.

Core Claim

The literal bias of contrastive vision-language models could arise at one of three stages, and each implies a different fix. At stage (a), the text encoder never computes a holistic meaning; the synergy residual is near zero, and only external knowledge helps. At stage (b), the residual exists but lies mostly outside the subspace spanned by image embeddings, so cosine similarity with images ignores it. At stage (c), the residual lies in the shared subspace but is swamped by the much larger literal component, so simply scaling it up should work.

We predict (b) for mid-sized encoders, with a shift toward (c) as encoder size grows. The synergy paper found that the residual emerges late in training and grows with model scale, which motivates this prediction. The practical consequence is an intervention called synergy transport. We learn a linear text-to-image map from ordinary literal caption data, apply it only to the synergy residual, and add the result back to the query.

This differs from all existing work in the file. The SemEval-2025 AdMIRe systems (AIMA, AlexUNLP-NB, PALI-NLP) route around the joint space with classifiers, paraphrasers or captioners. XMPIE measures the failure but does not explain it. The synergy calculus has only been applied to text-only embedders. The modality-gap literature studies a single global offset between modalities, not phrase-specific residual directions.

Measuring synergy in context and splitting it by subspace

Two of the three benchmarks put the target expression inside a sentence, and in AdMIRe that sentence can use it literally. We therefore compute synergy in context rather than on the bare phrase. Let $s$ be the sentence, let $a$ and $b$ be the two constituent words of the compound or idiom, and let $s_{\setminus a}$ denote $s$ with $a$ replaced by a mask or neutral filler. Let $T(\cdot)$ be the text encoder's output in the joint embedding space before normalization. The two-constituent Möbius residual is

$$q(s) = T(s) - T(s_{\setminus a}) - T(s_{\setminus b}) + T(s_{\setminus ab}).$$

The three-word version follows the span formula of the synergy paper. For a literal use in context, $q$ should be small, because removing each word subtracts roughly its own contribution. For a figurative use, $q$ should be large.

Next we estimate the image subspace. Let $P_U$ be the projection onto the top-$k$ principal directions of centered image embeddings, computed from a neutral image corpus (MS-COCO). We split the residual as $q = P_U q + (I - P_U)q$. The key diagnostic is the visible fraction $\rho = \|P_U q\|^2 / \|q\|^2$. We compare $\rho$ across idiomatic and literal uses and against the same ratio for ordinary caption-to-caption differences.

We also test where the figurative content sits. For each item we compute a text-side reference, the difference between the embedding of a gold paraphrase such as "ease social tension" and the literal phrase. We then measure the cosine of that reference with each half of $q$. We run the same test on the image side, using the difference between the idiomatic and literal candidate images. Hypothesis (b) predicts three things together: $q$ aligns with the paraphrase direction, most of that alignment comes from $(I-P_U)q$, and $P_U q$ is nearly uncorrelated with the image-side figurative direction.

The mechanism of the modality-gap papers motivates this subspace test. Mind the Gap shows that separately initialized encoders produce embeddings in narrow, differently oriented cones. Is the Modality Gap a Bug or a Feature? shows that contrastive training leaves a gap vector orthogonal to both modalities. Towards understanding the modality gap in CLIP reports that text and image embeddings are 100% linearly separable. Together these say text has room in directions images do not occupy. We ask whether idiomatic meaning is stored there.

Synergy transport

We fit a ridge-regression map $W$ from text embeddings to paired image embeddings on MS-COCO captions. The map therefore never sees an idiom. The steered query is

$$t' = T(s) + \alpha \, g(s) \, W q(s),$$

where $\alpha$ is a single global scale tuned on a development split. The gate $g(s)$ is the synergy norm $\|q(s)\|$ passed through a sigmoid fit on the development split, so literal uses are left nearly untouched.

The logic of applying $W$ only to $q$ is as follows. Mapping the whole query through $W$ would still be dominated by the literal words. Mapping only the residual carries its semantic content, which the synergy paper shows lands near words like "friendly" and "meet," into image directions that $W$ learned from literal captions of those concepts.

If the diagnostic instead finds stage (c), the right intervention is $W = I$, which amounts to plain amplification of the residual. That case is exactly the text-space steering the synergy paper proposed, and it becomes our result. If the diagnostic finds stage (a), transport cannot help. We then fall back to computing $q$ in a strong text-only embedder that is known to carry synergy (Qwen3-Embedding, used in the synergy paper) and fit $W$ from that embedder into CLIP space on the same COCO captions. That fallback is still a single embedding pass and needs no generator.

Research Program

The work proceeds in three stages on OpenAI CLIP ViT-L/14, a SigLIP model and EVA-CLIP-18B. This set spans text-encoder sizes, which lets us test whether the localization shifts from (b) toward (c) with scale.

The first stage is detection. Does $\|q\|$ separate idiomatic from literal uses in CLIP text encoders? We measure this on MAGPIE contexts (MAGPIE) and on the curated idiom pairs of the synergy paper, where Qwen-class text embedders reach ROC-AUC 0.90–0.92. We also compute the residual at intermediate text-encoder layers, because the synergy paper found the signal in middle-to-late layers of Pythia. It may be present internally but lost in CLIP's final pooled token.

The second stage is localization, using the visible-fraction and alignment diagnostics described above. We add one image-side sanity check: retrieval from the gold paraphrase to the idiomatic image. If paraphrases retrieve figurative images well, the image encoder is not the bottleneck.

The third stage is intervention. We run synergy transport on XMPIE and AdMIRe Subtask A, including the extended AdMIRe set, where compounds appear in their opposite sense.

Baselines

The naive reference point is zero-shot retrieval with the unmodified text query. On XMPIE English with EVA-CLIP-18B, this puts the idiomatic image first 6% of the time and the literal image first 90% of the time (A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding). We rerun it for every encoder so that all comparisons share the same embeddings.

The strongest competitor on our specific question is the paraphrase pipeline of AlexUNLP-NB. An instruction-tuned language model first classifies the usage as literal or idiomatic, then rewrites idiomatic uses into plain words, and the same CLIP ranks images against the rewrite. It is the strongest competitor because it tests the same thing we do: whether the right text vector fixes retrieval. It also ranked 3rd in English on AdMIRe Subtask A. No reported result was found on Lacuna for its Acc@1. We will therefore reimplement it with the same CLIP encoders, the same language model for all language-model steps, and the same development split for any threshold, and report its accuracy ourselves.

We treat the paraphrase pipeline as a soft ceiling for any method that keeps CLIP's text encoder as the only source of meaning. We count synergy transport as successful if it meets two conditions. First, it recovers at least half of the paraphrase pipeline's gain over zero-shot on idiomatic items. Second, it lowers the literal-first rate without reducing accuracy on literal-usage items, and it does this at roughly four encoder passes and no generation.

For absolute context on AdMIRe we report two further systems. The first is MIRA from PALI-NLP, which reaches Acc@1 0.93 on Subtask A English with text and images, against a best individual human annotator at 0.86 on the extended evaluation set (SemEval-2025 Task 1: AdMIRe). The second is AIMA, with Rank-1 accuracy of 65% on the test set and 50% on the extended set, and 87% and 48% for its caption-free variant. We do not claim to beat MIRA, which uses closed language models, captioning and repeated self-consistency passes. We do claim to beat AIMA's extended-set figure, since that set specifically tests sense reversal, which our context-dependent gate is designed to handle.

Four ablations isolate the proposal's ingredients. Global gap translation shifts the query by the mean text-to-image offset, in the style of the post-processing in Is the Modality Gap a Bug or a Feature?. Untransported steering adds $\alpha q$ directly in text space, as in the synergy paper. Whole-query transport maps $W T(s)$ instead of $W q(s)$. Naive subtraction replaces the Möbius residual with $T(s)$ minus its unigram embeddings, which the synergy paper found weaker at detection. All ablations share $W$, $\alpha$ search ranges and the development split.

Why not just…?

Why not just have a language model paraphrase the idiom and query CLIP with the paraphrase? This is the real threat, and it will probably score higher than synergy transport. It imports world knowledge that a caption-trained text encoder may lack. Two points keep the proposal worthwhile. First, the pipeline cascades errors from its detection stage and lost ground in Portuguese to direct multilingual retrieval (AlexUNLP-NB). Our method needs no generator in any language, and XMPIE's 34 languages let us test exactly where the paraphraser's advantage erodes. Second, the paraphrase pipeline says nothing about why CLIP fails. Our localization result stands whether or not transport wins. If transport recovers little of the pipeline's gain even under stage-(b) evidence, we will report that the residual carries too little content, which is itself a finding.

Why not just close the modality gap? Closing the Modality Gap Aligns Group-Wise Semantics shrinks the gap from 0.47 to 0.03 and doubles clustering quality on MS-COCO. It also reports that cross-modal R@1 stays stable, because retrieval depends on ranking rather than distance. That result predicts that removing a global offset will not reorder a literal and a figurative image for a fixed idiom query. Our gap-translation ablation, together with retrieval using a ModGap-style fine-tuned checkpoint if one can be trained on COCO within budget, tests this directly. A null result there supports our claim that the problem is a phrase-specific direction, not a mean shift.

Why not just fine-tune on idiom data, with a router, as AIMA did? AIMA's drop from 65% to 50% on the extended set, and from 87% to 48% for its caption-free variant, shows supervised routing overfitting when compounds appear in their other sense. Our $W$ is trained only on literal COCO captions, and our gate comes from the input's own residual. We also include a matched-budget baseline: a linear probe fine-tuned on AStitchInLanguageModels idiom pairs. If that probe beats transport on the extended set, cheap supervision wins, and we will say so.

Why not just caption the images and reason in text? MIRA does this best. However, it relies on elaborate in-context correction of language models' opposite bias toward figurative readings (PALI-NLP), and it needs many passes per item. It answers a different question from ours: it asks how to solve the task, not what the embedding space represents. We report it as context rather than as a competitor.

Proposed Experiments

The decisive checks come in order, and each can change the plan.

The first check is detection. If $\|q\|$ in CLIP text encoders fails to separate idiomatic from literal MAGPIE uses clearly above chance, stage (a) holds. We then switch to external-embedder transport and report that CLIP's literal bias originates in the text encoder, not the gap.

The second check is localization, and it is the main scientific result. We look for idiomatic residuals with a lower visible fraction $\rho$ than literal residuals and caption differences, and for the paraphrase-aligned content concentrated in $(I-P_U)q$. We report this across the three encoder sizes to test the scale prediction.

The third check is intervention. On XMPIE, in English and on held-out languages, we measure idiomatic-first rate and literal-first rate. On AdMIRe Subtask A, in English and Portuguese, we measure Acc@1 and DCG on both the test and extended sets. Throughout, synergy transport is compared with the naive query, the reimplemented paraphrase pipeline, and the four ablations.

The pattern that would confirm the proposal has three parts. Transport should beat untransported steering and gap translation. It should beat whole-query transport, which shows the residual carries the signal. And it should hold its gain on AdMIRe's extended set, where literal uses must stay literal.