Partial Matching of Object Slots to Captions as a Test of Why the Modality Gap Exists

Image–text models like CLIP place images and captions in one shared space, yet the two modalities sit in separate regions of that space, divided by a "modality gap." A recent explanation is that captions say less than images do. Captions name the main objects and omit most attributes and background things. Because a single image vector must still be matched to a thinner text vector, the model opens a gap along a few dimensions to absorb that mismatch. We propose to test this explanation by removing the mismatch in the architecture instead of in the data. We represent each image as a set of object "slots," which are vectors that an unsupervised model assigns to separate objects or regions of a scene without any labels. We then train the text side so that each noun phrase in a caption must find its slot, while slots the caption never mentions are allowed to stay unmatched at no cost. If information imbalance is what creates the gap, this partial matching should keep the gap closed even when captions are aggressively thinned. The same condition widens the gap in standard whole-image training.

In two sentences: captions describe only part of a scene, and forcing one image vector to agree with a partial description is believed to create the modality gap. We give the model separate vectors for each discovered object and score only the ones the caption mentions, then check whether the gap and the bias toward naming objects rather than their properties both disappear.

Core Claim and What Is New

The claim has two parts. The first is causal. When images are factorized into unsupervised object slots and the image–text score is one-directional, meaning every phrase must be explained but not every slot, the gap between matched slot embeddings and phrase embeddings should stay small and roughly constant as caption words are dropped. A whole-image model trained on the same thinned captions should show a growing gap. The second part is practical. Unmatched slots should keep the attribute and category information that the caption omitted, so the visual side continues to form categories for things language never named.

The motivating evidence comes from Two Effects, One Trigger. That paper shows that thinning captions on the synthetic Multi-modal Attributes and Digits dataset (MAD) and on CC12M widens the gap and increases object bias, that the gap concentrates in very few dimensions and acts as an entropy regulator, and that enriching captions shrinks both effects. It studied only whole-image embeddings, and it fixed the imbalance by adding text. We fix it by changing what the image side is asked to match.

Mind the Gap supplies a competing mechanism: the cone effect of deep encoders plus a repulsive low-temperature contrastive loss. Our design must separate that mechanism from imbalance. Discovering Hidden Visual Concepts Beyond Linguistic Input in Infant Learning shows that a monolithic encoder trained on infant head-camera video with caregiver speech still forms categories for unspoken objects (76.81% out-of-vocabulary accuracy on the Konkle objects). That result supports our second claim, but its encoder does not separate those objects into their own vectors.

The closest existing method is Improving Cross-Modal Retrieval with Set of Diverse Embeddings. It uses a Slot Attention–style module to produce sets of embeddings for both images and captions, aligned with a symmetric Smooth-Chamfer similarity. Our proposal differs in three concrete ways. First, its slots are trained only by the retrieval loss, so they are feature pools rather than discovered objects. Ours come from an unsupervised object-discovery model and are checked for mask quality. Second, its similarity is symmetric, so unmentioned image content still has to be pulled toward text. Ours is deliberately one-directional and includes an explicit "unmentioned" option. Third, it never measures the gap or object bias. We treat its symmetric set matching as the key ablation, not as prior art we merely extend.

Research Program

The slot source is an object-centric model trained without labels. Slot Attention (Object-Centric Learning with Slot Attention) is a module in which a fixed number of vectors compete, through attention, to explain parts of an image's feature map. We run it on frozen DINOv2 features with a feature-reconstruction objective, the same instance signal that GLASS uses. We add a background register slot in the style of SlotAdapt so that sky and road do not dilute object slots. We deliberately do not use GLASS's semantic guidance: it derives masks from captions via BLIP-2 and Stable Diffusion, which would reintroduce caption bias into the slots we are trying to test.

We train the slot model first and freeze it. This way the visual decomposition is never shaped by the captions. A projection head then maps slots into the joint space. Text is parsed into noun phrases, each encoded in context by a text transformer.

For an image with slots $s_1,\dots,s_K$ plus a learned null vector $s_\varnothing$, and a caption with phrase embeddings $z_1,\dots,z_P$, the image–text score is

$$S(I,T)=\frac{1}{P}\sum_{p=1}^{P}\tau\log\sum_{k\in\{1..K,\varnothing\}}\exp\!\left(\langle z_p,s_k\rangle/\tau\right),$$

where $\tau$ is a temperature and the log-sum-exp is a soft maximum over slots. $S$ enters a standard InfoNCE loss across the batch. Nothing in $S$ sums over slots, so a slot no phrase selects receives no gradient pulling it toward text. The null vector lets a phrase with no visual referent (for example, "a nice day") avoid corrupting a real slot.

The measurement program is as important as the model. For each trained system we compute the gap on matched pairs. For whole-image models this is the image vector versus the caption vector. For slot models it is the phrase vector versus its selected slot. We report it both as a mean-difference norm and as the relative modality gap (RMG) of Two Effects, One Trigger. We also count how many principal dimensions carry the gap, and measure object bias with the Matching Object Attribute Distance (MOAD) from the same paper.

To separate imbalance from the cone effect, we also measure the gap at initialization. Mind the Gap predicts a nonzero initial gap regardless of design. The imbalance account predicts that only the growth of the gap during training, and its sensitivity to caption thinning, should differ between designs.

Baselines

The naive reference point is a CLIP-style dual encoder on whole images, trained on the same image–caption pairs, backbone features, batch size and step budget. This is the setting in which Two Effects, One Trigger observed the gap widening under caption thinning. On real data we additionally report off-the-shelf CLIP. For zero-shot CIFAR-10, Mind the Gap reports top-1 accuracy of 0.9013 at CLIP's default gap and 0.9081 after tuning the gap. These are the numbers any gap intervention must not fall below on that task.

The strongest competing method is the set-of-embeddings model of Improving Cross-Modal Retrieval with Set of Diverse Embeddings. The file reports its gains only as relative RSUM improvements over the earlier set-based retrievers PVSE and PCME (+3.3 points on COCO 5K, +10.0 points on Flickr30K); no absolute RSUM is given on Lacuna. We therefore retrain it ourselves on identical data with the same backbone and an equal hyperparameter-search budget of 20 trials, tuning both systems' temperature and slot count. Beating it means two things. On retrieval, a matched or better RSUM on COCO and Flickr30K. On the gap question, a caption-thinning curve whose RMG and MOAD slopes are significantly flatter than its slopes across three seeds.

The data-side competitor is caption enrichment, the remedy Two Effects, One Trigger recommends. We implement it by training the whole-image model on captions rewritten to include attributes: generated attribute-complete captions on MAD, and dense model-generated captions on CC12M. It gets the same compute budget. For fine-grained grounding, the reference is MAF. It reports 61.43% phrase-grounding accuracy on Flickr30k Entities under weak supervision and 56.05% zero-shot, using supervised Faster R-CNN regions. We do not expect to beat a detector trained on labeled boxes outright. Coming within a few points while using no box labels would count as a meaningful result, and exceeding its 56.05% zero-shot figure would count as beating it.

Two ablations isolate the key ingredient. The first, symmetric matching, adds the reverse term so that every slot must also match some phrase. It uses exactly our slots and changes only the score, so it tests whether "unmentioned is allowed" is what matters. The second, patch groups, replaces the slots with k-means clusters of DINOv2 patch features, holding the count fixed. It tests whether object-level decomposition matters or any grouping would do. To show the frozen slots are genuine objects, we report their foreground Adjusted Rand Index (FG-ARI, agreement between predicted and ground-truth object masks) on COCO against SlotAdapt's reported 41.4.

Why not just…?

Why not just enrich the captions? This is the most serious threat, because Two Effects, One Trigger already shows it shrinks the gap and object bias. Our answer is partly practical and partly scientific. Enrichment requires text that often does not exist: caregiver speech in SAYCam-S cannot be rewritten, and generated captions carry their own captioner's object bias. More importantly, enrichment cannot tell us whether the gap is caused by the imbalance itself or by forcing a whole-image vector to absorb it. The experiment settles this directly. Our slot model is trained on 75%-thinned captions, and the whole-image model on enriched captions. If the enriched whole-image model matches our gap, MOAD and retrieval at comparable cost, then our method's value reduces to settings without rich text, and we will report it that way.

Why not just subtract the mean gap vector after training? Two Effects, One Trigger reports that post-hoc gap closing degrades downstream performance because the two sides have differently shaped neighborhoods. Extending Multi-modal Contrastive Representations finds that a single transformation cannot fix modality separation while preserving a base space's geometry. We still include mean-shift as a cheap control on every checkpoint. If shifting our whole-image baseline recovered the slot model's retrieval and MOAD, the architectural claim would be weakened.

Why not just align words to dense patches, with no slots at all? Word-to-patch attention in the style of SCAN also permits unmentioned regions to stay unmatched. The set-embedding paper reports roughly 80 times fewer inference FLOPs than SCAN for slot sets, which is an efficiency argument, not evidence about the gap. The honest test is our patch-group ablation, together with a max-over-patches version of our score. If patches match slots on the thinning curve, the conclusion is that partial matching, not object discovery, is the operative ingredient, and we will say so.

Why not just use a detector or a captioner to find objects? MAF needs Faster R-CNN regions from fixed classes. A captioner imports exactly the naming bias under study. Our slots are label-free, so any category they form beyond the caption vocabulary is attributable to vision alone.

Proposed Experiments

The controlled experiment uses a multi-object extension of MAD: two to four Morpho-MNIST digits per image, each with its own thickness, swelling, color and scale. Captions are thinned so that each caption mentions a random subset of digits and of their attributes. Because the ground truth is known, we can sweep the mention rate from 100% down to 25%. The decisive check is the slope of RMG and MOAD against mention rate for four systems: the whole-image model, symmetric set matching, patch groups, and our partial matching. The imbalance hypothesis predicts a clearly positive slope for the first two and a near-flat slope for ours. A positive slope for ours falsifies the claim that slot factorization with partial matching resolves imbalance. Equal slopes for ours and the patch-group ablation indicate that object discovery is not needed. We also linearly probe unmatched slots for the attributes of unmentioned digits, which should be decodable in our model and degraded in the symmetric one.

The real-data experiment repeats the thinning sweep (0%, 50%, 75% of words dropped) on CC12M as in Two Effects, One Trigger. We then evaluate COCO and Flickr30K retrieval (RSUM), Flickr30k Entities grounding by reading off the selected slot's mask, and zero-shot CIFAR-10 against the 0.9013 / 0.9081 reference. We report the initial-versus-trained gap for every system to separate the cone-effect contribution predicted by Mind the Gap from the imbalance-driven growth.

A final, smaller study trains the same pipeline on SAYCam-S caregiver speech. It asks whether unmatched slots cluster into Konkle categories the speech never names. The comparison point is the 76.81% out-of-vocabulary accuracy reported for the monolithic encoder in Discovering Hidden Visual Concepts Beyond Linguistic Input in Infant Learning, with the caveat that the evaluation protocol there uses neuron-level probes, and we will rerun that protocol on both models rather than compare across protocols.