TL;DR
Contrastive vision-language models like CLIP suffer from two distinct quirks: a geometric "modality gap" that separates image and text embeddings, and a strong bias toward recognizing objects over attributes. This paper reveals that both phenomena share a single root cause: information imbalance, where rich images are paired with sparse, incomplete captions. By understanding this link, researchers can better mitigate these biases through targeted data filtering and caption enrichment.
What Problem Does This Paper Address?
Contrastive Vision-Language Models (VLMs) are incredibly powerful for zero-shot tasks, but they have a glaring weakness: they excel at identifying objects (e.g., "cat") but struggle with attributes (e.g., "striped" or "yellow"). Prior literature has frequently blamed this on two distinct phenomena:
- The Modality Gap: A geometric quirk where image and text embeddings occupy completely separate regions of the shared vector space.
- Object Bias: The tendency of VLMs to prioritize object-level features over other contextual factors.
However, it has remained unclear why these phenomena emerge in the first place, whether they actually harm downstream performance, and if they are bugs or inherent features of contrastive learning. This paper systematically investigates these questions to uncover the underlying mechanics connecting the two effects.
Methods
The authors approach the problem through a mix of large-scale empirical analysis and tightly controlled interventions:
- New Metrics: They introduce the Relative Modality Gap (RMG) to measure embedding separation more accurately than standard L2 distance-to-means (L2M), which fails to account for per-sample pairings. They also introduce Matching Object Attribute Distance (MOAD) (and its generalized form, BRACE) to formally quantify object vs. attribute bias.
- Dimensionality Ablation: They analyze the geometry of the modality gap by progressively zeroing out embedding dimensions to isolate which ones drive the separation.
- Controlled Synthetic Testing: They build a synthetic framework to systematically vary the amount of information present in text captions while keeping the visual data constant.
- Real-world Caption Manipulation: They train models on real-world data while artificially truncating captions to simulate varying levels of information scarcity.
Datasets
- Evaluation Benchmarks: MS COCO (retrieval), ImageNet (zero-shot classification), MIT-States, and UT-Zappos (fine-grained attribute/object classification).
- Pre-trained Models: 98 off-the-shelf VLMs (CLIP, SigLIP, etc.) trained on massive datasets like LAION-400M, LAION-2B, WebLI, and DataComp.
- Multi-modal Attributes and Digits (MAD): A new, fully-controllable synthetic dataset introduced by the authors. Based on Morpho-MNIST, it maps digits to objects and applies controlled morphing operations (thickness, swelling, fractures, scaling, color) as attributes to study information imbalance cleanly.
Key Insights and Ideas
- Information Imbalance is the "Trigger": Images naturally contain vastly more information than a short sentence can capture. Because an image encoder cannot guess which details a human decided to write down, it faces massive uncertainty. To minimize contrastive loss, the model prioritizes the most predictable elements (objects) and pushes the modalities apart to maximize uniformity.
- The Modality Gap is Driven by Just a Few Dimensions: Surprisingly, the entire geometric separation between text and images is driven by only a few embedding dimensions. In fact, just two dimensions are often enough to perfectly separate the two modalities.
- Closing the Gap Post-Hoc is a Bad Idea: Simply shifting or removing the gap-driving dimensions post-hoc does not improve downstream performance. This is because the local neighborhood structures of the image and text spaces are fundamentally different; they don't map 1:1.
- The Gap is a Feature, Not a Bug: The modality gap provides the model with the mathematical flexibility to manage logit entropy (uncertainty). Forcing the gap closed during training without fixing the underlying data imbalance actually hurts the model's performance.
- Object Bias is Actually "Caption Presence Bias": Models aren't biased toward objects because object words appear more frequently overall in a dataset (in fact, attributes often have higher global frequencies). Instead, it's driven by conditional probability: when an object is in an image, humans almost always name it, whereas they only selectively name its attributes.
Key Results
- Confounder Correction: While a naive look at off-the-shelf models suggests a larger modality gap correlates with better performance, controlling for dataset and model size reveals the opposite: a smaller modality gap actually yields better downstream results.
- Information vs. Gap: When training on the synthetic MAD dataset, increasing the number of attributes in the caption (reducing information imbalance) drastically shrinks the modality gap and object bias while boosting downstream accuracy.
- Real-World Validation: On CC12M, models trained on "full captions" exhibited a significantly smaller modality gap and better retrieval performance ($R@1$) compared to models trained on "half" or "quarter" captions.
- Fine-Tuning on Dense Data: Fine-tuning CLIP and SigLIP on the highly-aligned Densely Captioned Images (DCI) dataset successfully reduced the modality gap (lowering both L2M and RMG metrics).
Limitations and Future Work
- Compute Constraints: Due to resource limitations, the authors could not train models from scratch on massive, web-scale datasets like LAION; validation of the training hypotheses was restricted to small-to-medium datasets (MAD, CC3M, CC12M).
- Crude Imbalance Simulation: Dropping contiguous chunks of text (halves/quarters) to simulate information imbalance on real data is a rough heuristic and may introduce high variance.
- Future Directions: The authors suggest exploring techniques from partial multi-label learning to help VLMs handle missing caption information more gracefully without developing geometric gaps or concept biases.
Summary updated