Contrastive vision-language models such as CLIP embed an image and a caption as one vector each and score them by similarity. They cannot tell "a dog bites a man" from "a man bites a dog." On Winoground, a benchmark of caption pairs that use the same words in swapped roles, large models score at or below chance. Probing work suggests the language side is not the culprit, because CLIP's text encoder separates the swapped captions well. The failure lies in how roles are tied to image regions. Frame-based knowledge representation was built for exactly this binding problem. In that tradition, a situation is a template with named slots (Agent, Patient, Tool, Place), and each slot is filled by a specific participant. We propose to put that structure back into the scoring rule. A caption is parsed into a frame. Each slot gets its own role-conditioned query that pools the frozen image encoder's patch features into one vector. The image-caption score is then a sum of slot-by-slot matches rather than a single global cosine.
The two-sentence version is this. Today's models compare whole images with whole sentences, so swapping who does what to whom barely changes the score. We instead compare each frame slot of the sentence with the image region that plays that role, and we share slot queries across verbs through FrameNet-style role inheritance, so the structure should carry over to verbs never seen in training.
Core Claim
Role-swap failures are a read-out problem, not a geometry-of-the-gap problem. The "modality gap" refers to the observation that image and text embeddings occupy separate cones of the shared space. Mind the Gap shows this gap comes from separate encoder "funnels" and contrastive temperature, and that shifting one modality toward the other can modestly help or hurt downstream tasks. A shift of that kind cannot fix role swaps, for a simple reason. Two swapped captions have nearly identical text embeddings, and a uniform translation moves both by the same vector, so their ranking against a given image barely changes. Closing the gap globally, as Mitigate the Gap (AlignCLIP) does with shared top layers, improves classification. Nothing in that work targets role binding. We claim the missing ingredient is a role-indexed decomposition of the score.
We define the score of image $I$ against caption $C$ as
$$ s(I,C) = \lambda\, \cos\big(v(I), t(C)\big) + (1-\lambda) \frac{1}{|R_C|}\sum_{r \in R_C} \cos\big(g_r(I),\, t(f_r)\big). $$
The terms are as follows. $v$ and $t$ are the frozen CLIP image and text encoders. $R_C$ is the set of roles filled in the frame parsed from $C$. $f_r$ is the text filler of role $r$ (for example "the man"). $g_r(I)$ is an attention-pooled vector computed from CLIP patch tokens by a learned query for role $r$, conditioned on the frame's verb embedding. $\lambda$ trades off the global and slot terms.
Swapping fillers changes which filler is compared with which role-pooled region. The slot term is therefore sensitive to swaps by construction, and it is only correct if $g_r$ actually finds the right participant.
Relation to Prior Work
Three works motivate the proposal by locating the failure. Winoground built 400 minimal pairs and reports group scores of 14.50% for VinVL and 14.25% for FLAVA, against 16.67% chance and 85.5% for humans. Both models already use cross-attention fusion, which matters for the alternatives discussed below. Why is Winoground Hard? shows that CLIP's text side distinguishes caption variants above 80%, while CLIP succeeds on 0% of the "visually difficult" items. That pins the problem on the cross-modal step and partly on resolution. We take this as a constraint: some Winoground items will stay out of reach for any read-out over a frozen low-resolution encoder, and we report those items separately.
Three works supply the frame structure and the supervision. Situation Recognition imported FrameNet-style verb frames into vision as a closed-vocabulary conditional random field over CNN features. Grounded Situation Recognition added SWiG's bounding boxes and a sequential verb→noun→box predictor. CoFormer made that predictor bidirectional and reaches 44.41% top-1 verb accuracy and 29.37% grounded value accuracy on SWiG. All three are classifiers over a fixed list of 504 verbs and about 11,000 nouns. We reuse their annotations as supervision for slot queries, but we never classify. Fillers are open text scored in CLIP's space, and the target task is image-caption matching, not situation prediction. The Berkeley FrameNet Project supplies frame inheritance. Role queries are tied across verbs that inherit a frame element such as Agent, and this tying is what we test for transfer to unseen verbs.
Several works are neighbors that we deliberately differ from. APT makes frozen language embeddings context-sensitive with visually conditioned prompts, but only for pairwise scene-graph edges. A frame instead binds several roles at once under one verb. MAF grounds isolated noun phrases with attention, with no notion of role. Extending Multi-modal Contrastive Representations supports our design choice of keeping a pretrained space frozen and adding small heads. Its finding that one projector cannot serve two alignment goals is why we keep the global and slot paths separate rather than retraining one projection.
Research Program
The system has three parts. The first is a frame parser that turns a caption into a verb plus role-filler pairs. We use template verbalizations for SWiG training data, and a semantic role labeler with a FrameNet-to-imSitu role mapping for free text. Parses of the Winoground evaluation captions will be hand-checked.
The second part is the set of role queries $g_r$. Each is a single cross-attention layer over frozen CLIP patch tokens, with one learned query per role. The query is modulated by the verb's text embedding, so that "Agent of cutting" and "Agent of riding" can differ while still sharing parameters.
The third part is the training objective on SWiG, which has two terms. One is a slot-level contrastive loss in which each pooled vector $g_r(I)$ must match its own filler's text embedding against the fillers of the other roles in the same image. The other is an auxiliary term that pushes the attention map toward the annotated box when one exists.
The within-image contrast is the frame-specific part. A filler is told apart from co-occurring participants playing other roles, not merely from random captions.
The backbone stays frozen, so the global path and CLIP's zero-shot behavior are unchanged. Everything new is attributable to the read-out. We also measure a slot-level analogue of the modality gap: the distance between the centroid of $g_r(I)$ vectors and the centroid of filler embeddings, computed per role. We test whether roles with larger residual gaps are the ones bound least accurately. This links Gap 2 of the research file to a measurable quantity instead of a slogan.
Baselines
The naive reference point is frozen CLIP with its global cosine, evaluated on exactly the same images and captions. CLIP's group score on Winoground is not reported in the file ("no reported result found on Lacuna for CLIP Winoground group score"), so we rerun it ourselves. The cited fusion-model numbers, 14.50% for VinVL and 14.25% for FLAVA, serve as context. Beating the naive point means exceeding the 16.67% chance level on the group score by a margin larger than the bootstrap interval over the 400 items, and improving the text score on our SWiG role-swap test.
The geometric baseline is CLIP with the mean-shift gap correction from Mind the Gap: the text centroid is translated toward the image centroid, with the shift magnitude tuned on a SWiG validation split. We predict this baseline will tie frozen CLIP on role swaps, and confirming that prediction is itself a result. Where an AlignCLIP-style shared-layer checkpoint can be trained on the same data budget, we include it as the stronger gap-closing variant. Its reported figures are a 0.47→0.67 rise in alignment score and +8% on CIFAR-100 zero-shot (Mitigate the Gap). None of those figures concern role binding.
The strongest competitor on the actual question is hard-negative fine-tuning. We fine-tune CLIP's last transformer blocks contrastively on SWiG-verbalized captions, with each image's caption paired against a role-swapped version of itself as a negative. This baseline uses the same images, the same frame annotations to generate the swaps, the same GPU-hours, and the same hyperparameter search budget as our method. It is the strongest because it uses exactly the same supervision signal without the slot decomposition. To beat it, our method must win on the held-out-verb split and on Winoground, not just on in-distribution swaps.
For grounding we compare against CoFormer's reported 29.37% grounded value accuracy on SWiG (CoFormer). We do not expect a frozen-backbone read-out to beat a fully trained detector there. Staying within a few points of it would show the slots really localize participants.
Three ablations isolate the frame ingredient. The first replaces role-conditioned queries with role-agnostic ones (the same number of slots, matched to fillers by Hungarian assignment). The second unties queries across verbs, which removes FrameNet inheritance. The third drops the box term to test whether role contrast alone suffices.
Why not just…?
Why not just fine-tune CLIP with role-swapped hard negatives? This is the real threat, and the experiments are built to settle it. Hard negatives teach the global embedding to register swaps it has seen, but nothing forces it to locate the participants. Our prediction is that hard negatives match the slot read-out on swaps involving training verbs, and fall behind on held-out verbs and on Winoground's free-form captions, where the slot read-out can reuse shared Agent and Patient queries. If hard negatives match us on the held-out-verb split too, the honest conclusion is that frame structure adds interpretability and localization but not accuracy. We would report it that way.
Why not just scale the pretraining data? Winoground evaluated models trained on hundreds of millions of pairs and found chance-level group scores. Why is Winoground Hard? shows the text side already encodes the distinction. Scale has not supplied the binding step, and our frozen-backbone design asks whether a small structured head can supply it without more data.
Why not just close the modality gap? As argued above, a uniform shift cannot reorder two near-identical caption embeddings. The mean-shift baseline tests this directly. AlignCLIP-style closure changes the encoder itself, so it could in principle help, and we include it where budget allows. Its published gains are on classification and robustness, not on compositional pairs.
Why not just use cross-attention fusion or pairwise scene graphs? VinVL and FLAVA already fuse modalities with cross-attention and still score 14.50% and 14.25% on the Winoground group score. Fusion without role indexing does not bind. Scene-graph methods such as APT score subject-predicate-object edges one at a time, and Visual Genome reports only about 8.7% baseline relationship accuracy. Neither represents a verb that constrains three or four participants jointly, which is what SWiG frames annotate.
Proposed Experiments
The first decisive check is a SWiG role-swap test. For test images whose frames have at least two filled roles, the true verbalized caption competes against captions with fillers permuted across roles, and we report text-score accuracy. We run it on two splits: one with verbs seen in training, and one with about a fifth of the verbs held out entirely, chosen so that their roles are inherited from FrameNet parents present in training.
The second check is Winoground. We report text, image, and group scores on all 400 items, falling back to the global score when a caption has no verb frame. We also report results on the subset whose captions parse to a frame with at least two roles, and separately on the "visually difficult" items, where we expect no method over a frozen encoder to succeed.
The third check is SWiG grounding. We report value and grounded value accuracy when the attended region of $g_r$ serves as the box prediction, to confirm that slot gains come from correct localization rather than shortcut cues.
The fourth check is the per-role gap analysis. We correlate each role's slot-level centroid distance with its binding accuracy.
We will count the idea as working if three things hold together. The slot read-out must beat the matched hard-negative baseline on the held-out-verb swap test. The role-agnostic ablation must lose most of that margin. The mean-shift baseline must stay near frozen CLIP. If the role-agnostic ablation does as well as the full method, the benefit comes from region pooling, not from frames, and the proposal's central claim fails.
References
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
- Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality
- Situation Recognition: Visual Semantic Role Labeling for Image Understanding
- Grounded Situation Recognition
- Collaborative Transformers for Grounded Situation Recognition
- The Berkeley FrameNet Project
- APT: Towards Universal Scene Graph Generation via Plug-in Adaptive Prompt Tuning
- MAF: Multimodal Alignment Framework for Weakly-Supervised Phrase Grounding
- Extending Multi-modal Contrastive Representations
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations