Teaching CLIP Who Did What to Whom with Generated Swap Quadruplets

Vision-language models such as CLIP match images to captions well but often treat a caption as an unordered bag of words. They can tell that a photo contains a red cup and a blue plate. They cannot reliably tell it apart from a photo of a blue cup and a red plate. Recent work generates synthetic training images to repair this, but it only changes one thing at a time, such as recoloring one object or replacing one noun. Those are exactly the cases that plain word matching already handles.

We propose generating swap quadruplets: two images and two captions that use identical words, where the only difference is which object gets which attribute or which spatial role. Each image is checked by an object detector before it is used for training. Such pairs are invisible by construction to any model that adds up word meanings, so they carry training signal only about binding and role structure. Because they come with two images per caption, they also teach the image side of the model: given one caption, which of two nearly identical scenes fits it? Text-only hard negatives cannot supply that signal.

Core Claim

The claim has three parts. First, a model trained on detector-verified, scene-matched swap quadruplets for attribute binding and spatial relations should gain substantially more on swap-type evaluations than single-edit synthetic data. Second, it should gain more than fluent text-only swap negatives. Third, the gain should show up specifically in the image-side scores that text-only negatives cannot train.

The central empirical question is whether generated images are needed at all for swap compositionality, or whether better text negatives suffice. We design the study so that it can answer "no".

Why This Intersection, and What Is Already Done

Compositional distributional semantics, the study of how phrase meaning is built from word vectors, gives the precise diagnosis. It has long been noted that additive composition cannot distinguish "Peter kicked the ball" from "The ball kicked Peter" (Giesbrecht). Other work in this line models modifiers and verbs as asymmetric operators, such as adjective matrices (Baroni et al.) or verb tensors (Polajnar et al.), precisely to recover argument roles. This literature motivates our design: the quantity to teach is role asymmetry, and a swap pair isolates it perfectly because the two captions share one bag of words.

We do not import tensor operators into the model. A reviewer would rightly ask whether a new architecture is needed when the failure may lie in the training data. Our proposal tests the data explanation first.

Vision-language benchmarks show that swaps are where models fail. On SugarCrepe, which removes text-only shortcuts by adversarial filtering (SugarCrepe), even GPT-4V drops from 90.95% overall to 83.13% on swap items. On SUGARCREPE++, most vision-language models including CLIP score 45–55% on swap tasks, against 96–100% for humans (SUGARCREPE++).

Two diagnoses conflict. Why is Winoground Hard? finds that CLIP's text representations separate permuted captions with over 80% accuracy and locates the failure in cross-modal matching. SUGARCREPE++, by contrast, names the text encoder as the main bottleneck. Our ablations are built to arbitrate between them.

Three existing methods connect generative data to compositionality, and our work builds on each:

  • SPARCL (Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data) is the closest prior work. It uses a language model to edit real captions, generates edited images while injecting the original image's global feature to preserve the scene, and down-weights failed generations with an adaptive margin. It reports over 8% average gain over base CLIP. By its own account it covers single attribute or object modifications, not multi-object role swaps. We supply the missing pair type and replace its soft down-weighting with explicit verification.
  • SDS-CLIP (Augmenting CLIP with Improved Visio-Linguistic Reasoning) distills a diffusion model's denoising signal into CLIP, with gains of up to 7% on Winoground. It inherits the teacher's binding flaws, which our verification step is meant to avoid.
  • StructureDiffusion (Training-Free Structured Diffusion Guidance) shows that the text encoder's causal attention blends adjacent noun phrases, and it fixes generation at inference time by encoding each noun phrase separately. We reuse it as a generator component to raise the binding success rate of our synthetic images.

In text-only compositional generalization, GECA and ARCHER show that recombining observed fragments helps a sequence-to-sequence model generalize. ARCHER also documents a constraint that we must face: strict filtering improves correctness but cuts diversity.

Research Program

We first build a scene-specification sampler from MS-COCO 2014 training captions. For each caption we parse out two entities with attributes, plus a spatial relation where one exists. We then write the swapped specification: attributes exchanged across entities, or subject and object exchanged in the relation. A language model produces two fluent captions per image, the original and a paraphrase, following SUGARCREPE++'s design so that models are also trained on lexical invariance. Scope is restricted to swaps a detector can verify, namely color and material binding and left/right/above/below relations. Action relations such as "chasing" are excluded because we cannot check them automatically.

Next comes generation. GenEval reports that current text-to-image models reach about 35% on attribute binding and about 15% on spatial position (GenEval). If the two images of a pair were generated independently, their failures would compound. We therefore generate the first image with StructureDiffusion-style guidance and verify it. We then produce its partner by editing only the detected entity regions, which recolors them or exchanges their boxes and inpaints the background. The background, lighting, and style stay fixed, in the spirit of SPARCL's scene preservation.

Verification follows GenEval's recipe, using Mask2Former detection for presence and position and a color classifier for attributes. We use a detector rather than CLIP for this step to avoid grading CLIP's training data with CLIP.

Training fine-tunes CLIP with a contrastive loss over each quadruplet in both directions. Each caption must prefer its own image, and each image must prefer its own caption. Paraphrases are added as extra positives. Throughout, we log acceptance rates per swap type to quantify the correctness–diversity trade-off that ARCHER identified.

Finally, an analysis step tests the arbitration question directly. We measure how far the fine-tuned text encoder separates swapped captions relative to paraphrases, which is text-side change. We also measure how far image embeddings of the partner images move apart, which is image-side change.

Baselines

The naive reference is off-the-shelf CLIP, together with CLIP fine-tuned on the same COCO pairs with the standard contrastive loss and no negatives. The second model separates the effect of our data from the effect of extra fine-tuning.

The strongest existing method is SPARCL, which reports "over 8%" average accuracy gain over base CLIP across ARO, VL-CheckList and SugarCrepe, and a 2% relative gain over prior best methods on three of four benchmarks. We will rerun it from its description on the same backbone, COCO-2014 training split, number of synthetic pairs, and optimizer steps. We will also quote its published numbers, noting that they may not be exactly comparable. Beating it means a higher score on SugarCrepe's swap-attribute and swap-object subsets and the SUGARCREPE++ swap tasks, with no loss on its replace subsets.

SDS-CLIP is a second generative-teacher competitor. It reports up to 7% absolute gain on Winoground. We train it with its LayerNorm-only recipe on the same COCO data. We also add a compute-matched variant of our method that updates only LayerNorm parameters, so that parameter count is not a confound.

The text-only swap-negative baseline is the most dangerous one. It uses the same language-model-written swapped captions as hard negatives for real COCO images, with no generated images. It shares our text pipeline, so the only difference is the synthetic image partner.

For spatial left/right swaps we add a horizontal-flip baseline. Mirroring a real image exchanges left and right for free, so pairing a flipped image with the swapped caption costs no generation. We will also try cut-and-paste region swapping on real images for the other relation types.

Two ablations isolate our key ingredients. One replaces detector verification with SPARCL's adaptive margin. The other generates the two images of a pair independently rather than by region editing.

Every system gets the same number of training pairs and gradient steps, and each receives the same small hyperparameter search on a held-out split of our own synthetic data, never on the test benchmarks.

Why not just…?

Why not just use fluent text-only swap negatives? This is the real threat. If SUGARCREPE++ is right that the text encoder is the bottleneck, text negatives may capture most of the gain on caption-ranking tasks. They cannot, however, train the image-to-caption direction, because they never show the model two images that differ only in role assignment. The decisive comparison is therefore Winoground's image score, and the group score that requires both directions. If text-only negatives match us there as well, we will conclude that generated images add nothing for swap compositionality and report that.

Why not just scale the model? The file's evidence argues against scaling alone. VinVL (14.50%) and FLAVA (14.25%) fall below Winoground's 16.67% chance level on group score (Winoground). GPT-4V still loses about 8 points on SugarCrepe swaps. We will nonetheless report the largest CLIP backbone we can afford, both untuned and with our data, to show whether the gain shrinks with scale.

Why not just flip or cut-and-paste real images? Flipping solves left/right and may beat us on that subset. We expect that, and we will report it. It cannot produce attribute swaps or above/below swaps, and cut-and-paste leaves seams that the model can use as a shortcut. Per-subset results will show where generation is actually needed.

Why not rule-based negatives like NegCLIP? SugarCrepe shows that such gains mostly came from exploiting ungrammatical distractors and vanished once those were filtered out. Our captions are written by a language model, and we evaluate only on adversarially filtered benchmarks.

Proposed Experiments

We train on quadruplets built from COCO-2014 training captions. Evaluation uses benchmarks derived from COCO validation data (SugarCrepe, SUGARCREPE++), plus out-of-domain Winoground and ARO. We check for overlap of captions and images with the test sets before training.

Metrics are SugarCrepe accuracy per subset, SUGARCREPE++ three-way hit accuracy per subset, Winoground text, image, and group scores, and ARO relation and attribute accuracy. We also track zero-shot classification to detect forgetting.

The decisive checks are three. First, we must beat the text-only swap-negative baseline on Winoground image score, not only text score. Second, we must beat SPARCL on the swap subsets while staying level on the replace subsets. Third, the acceptance-rate curves must show that region editing keeps enough accepted swaps per relation type to avoid the diversity collapse ARCHER reported. If the first check fails, the paper's conclusion becomes a negative result about image synthesis for this problem, which is still worth publishing.