Swap-Cloze Masking: Making Masked Word Prediction Depend on Who Does What in the Image

Vision-language models trained by filling in masked words can often fill them in without looking at the image. The relations and word order that distinguish "the mug is in the grass" from "the grass is in the mug" are the clearest case. Masked language modeling (MLM) hides random words and trains the model to predict them from the words around them. A multimodal version adds cross-modal attention, where text tokens query image patches for context. When a relation or argument word is masked, however, sentence plausibility usually suffices to recover it, so the loss gives cross-modal attention little reason to bind objects to roles. The symptoms appear on Winoground, a benchmark of image-caption pairs that differ only in word order. Strong multimodal models score below the 16.67% chance level on its Group Score (VinVL 14.50%, FLAVA 14.25%, against 85.5% for humans; Winoground). The text encoders alone can tell the two captions apart more than 80% of the time, which points to a failure in fusion rather than in language understanding (Why is Winoground Hard?).

We propose a change to the pre-training objective, not a fine-tuning patch. For captions with two arguments linked by a relation, we mask both argument slots together. We then preferentially choose the captions where the model's own text-only prediction cannot tell the true ordering from the swapped one, or prefers the wrong one. On these examples the image is the only source of the answer. The model therefore has to route role information through cross-modal attention during pre-training, at the token positions where binding happens.

Core Claim

Existing masked vision-language pre-training puts almost no masked-word loss on tokens whose correct filling requires visual role binding. Concentrating the loss on such tokens should do two things. It should close the fusion gap identified by Why is Winoground Hard?, and it should improve compositional matching more than two alternatives: changing the attention structure after the fact, or adding sentence-level hard negatives. We also claim that the effect is measurable token by token. Removing the image should raise the model's loss on argument and relation tokens far more after swap-cloze training than after random masking.

What Exists and Where This Differs

The masking literature establishes the setting. UNITER masks one modality at a time over detector region features. MaskVLM moves to raw pixels and jointly reconstructs masked text and masked image patches through a 3-layer cross-modal encoder. Its text and image masks are random and uncoordinated. SemMIM lets text choose which image patches to mask, but it leaves text masking random. XLIP masks named medical entities with an external tagger. That method targets content words, not order or role, and never asks whether language alone could already predict the masked span.

All four choose masks by what is salient. We choose masks by what language cannot resolve. That selection criterion, which exposes the missing image-to-role binding, is absent from all of them.

Two diagnostic papers motivate the design. Frank et al. introduced cross-modal input ablation, which removes one modality and measures the loss increase. With it they showed that removing images raises MLM loss by up to 25% while removing text raises region-classification loss by only 1–10%. We reuse their ablation in two ways. As a training signal, it computes our selection score. As an evaluation tool, it is applied per token category and to a raw-pixel model, which they did not test.

Cross-modal Attention Congruence Regularization (CACR) is the closest method aimed at the same failure. It fine-tunes UNITER's final attention layer so that text-to-text attention agrees with image-to-image attention projected through the cross-modal attention. This gains 5.75 Winoground Group points. CACR constrains attention shape but leaves the prediction objective untouched. Our proposal changes what the model must predict and leaves attention shape unconstrained, and the two can be compared directly.

Zhang and Hashimoto show that MLM's statistical dependencies track syntax and call for dependency-targeted masking. We treat their conditional-mutual-information parse as a detector-free alternative to a supervised parser for finding argument pairs.

Research Program

The base model follows MaskVLM: raw-pixel and text unimodal encoders, a cross-modal encoder, and losses for masked words, masked image patches, image-text contrast, and image-text matching. It is trained on the standard 4M corpus of COCO, Visual Genome, Conceptual Captions and SBU.

For each caption we extract argument pairs (A, relation, B) with a dependency parser. As an ablation we use the unsupervised parse from Zhang and Hashimoto. A swap-cloze example masks both A and B while keeping the relation word visible. The model reconstructs them jointly under the standard MLM loss.

Selection uses the model itself with the image removed. Let $\ell_T(x)$ be the masked-word loss of filling caption $x$ when image tokens are dropped from cross-attention, and let $x^{\text{swap}}$ be the same caption with A and B exchanged. The order-ambiguity score is

$$a(x) = \ell_T(x^{\text{swap}}) - \ell_T(x),$$

which is large when language alone already prefers the true order and near zero or negative when it does not. We sample swap-cloze examples with probability proportional to $\exp(-a(x)/\tau)$, where $\tau$ is a temperature. Counter-prior captions, such as a mug containing grass, are therefore oversampled. The score is refreshed from the current model every few thousand steps, so selection tracks what the model has not yet learned.

A fixed fraction $\rho$ of text masks, starting at 15% of MLM examples, comes from swap cloze. The rest remain random, which preserves general language learning. The added cost is one extra text-only forward pass on candidate captions, roughly a 10–20% increase in compute. All baselines receive the same increase.

Baselines

The naive reference point is MaskVLM-style pre-training with random masking. It uses the same architecture, the same 4M data, and the same steps plus 15% extra steps to match our compute. MaskVLM reports 60.1% MSCOCO 5K image-retrieval Recall@1, 75.45% on VQA and 81.98% on NLVR2. Our reproduction must land near these numbers before any comparison counts. No Winoground result was found on Lacuna for MaskVLM, so we measure it ourselves.

The strongest existing method on this question is CACR (Pandey et al.). The file reports only its +5.75 Group-point gain over UNITER. No reported absolute Winoground score for CACR or its UNITER baseline was found on Lacuna. We therefore run CACR ourselves in two ways, both with a matched tuning budget. The first is fine-tuning on top of the random-masking model. The second is an auxiliary loss added throughout pre-training, which is the obvious extension its authors left open. Beating CACR means a higher Winoground Group Score than both variants, with a bootstrap 95% interval over the 400 Winoground items that excludes zero difference. MSCOCO retrieval must stay within one point.

We also run SemMIM-style text-guided image masking as a salience-based masking competitor. Its reported numbers are 64.2 MSCOCO image-retrieval R@1 and 78.12% on VQA. A compute-matched reimplementation uses the same backbone.

Three ablations isolate the key ingredient. Joint argument masking without ambiguity selection, with uniform sampling over parsed pairs, tests whether the selection score matters. Ambiguity-selected masking of single random tokens tests whether the joint two-slot structure matters. Heuristic masking of all nouns and prepositions, analogous to XLIP's entity masking, tests whether ambiguity-guided masking is anything more than a masking rule based on part of speech.

Why not just…?

Why not just add swapped captions as hard negatives for image-text matching? This is the real threat, because it uses the same parsed pairs and costs less. It gives one bit of supervision per caption through the pooled matching token. The model can satisfy that bit with a global cue without binding each noun's token to its image region. We run the baseline with identical pairs and identical sampling by $a(x)$. The two can be told apart on two measures. The first is per-token cross-modal ablation on argument tokens: we predict swap cloze raises image dependence there and hard negatives barely do. The second is the Winoground Image Score, which requires grounding a caption in the correct image rather than ranking text. If hard negatives match us on both, we will report that the counterfactual pairs, not the masked-prediction framing, drive the gain.

Why not just apply CACR, possibly during pre-training? CACR aligns attention patterns but never requires that the aligned attention carry information the loss depends on. A model can make its attention congruent while still predicting from language priors. Our design runs CACR as a pre-training loss alongside our method and alone. If CACR alone matches us, the conclusion is that attention structure, not prediction targets, is the lever. We would then drop our claim.

Why not just use a contrastive dual encoder at larger scale? The evidence already argues against this. CLIP drops to 0% on the visually difficult Winoground items even though its text encoder separates the captions (Why is Winoground Hard?). Masked cross-attention models at 4M scale outperform much larger contrastive systems on retrieval (MaskVLM). We still report a CLIP-style dual encoder trained on the same 4M data with swapped-caption negatives as a reference.

Proposed Experiments

The first experiment is the main comparison. Every method above is pre-trained on the 4M corpus at matched compute, with three seeds per method at a reduced 1M-pair scale and one seed at full scale. We report Winoground Text, Image and Group Scores, MSCOCO and Flickr30k retrieval Recall@1, VQA v2 and NLVR2. The general-task numbers test whether swap cloze costs anything elsewhere.

Winoground is small, so we also build a held-out swap test from Visual Genome relation annotations on images excluded from pre-training. Each item pairs an image with its true and swapped caption. We report it separately for prior-consistent and counter-prior relations. A method that only learns plausibility will look good on the first split and fail on the second.

The mechanism check applies Frank et al.'s input ablation per token category (argument nouns, relation words, other words) to the base model, the hard-negative model and ours. The decisive prediction is that swap cloze raises vision-for-language loss increases on argument tokens well above the random-masking model, with little change on other tokens. If Winoground improves while this per-token dependence does not, the gain comes from somewhere other than the proposed mechanism. We also sweep $\rho$ and $\tau$. We compare the supervised parser against the unsupervised MLM parse, to test whether the method needs an external parser at all.