Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models

Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, Thomas Brox

TL;DR

Contrastive vision-language models like CLIP suffer from two distinct quirks: a geometric "modality gap" that separates image and text embeddings, and a strong bias toward recognizing objects over attributes. This paper reveals that both phenomena share a single root cause: information imbalance, where rich images are paired with sparse, incomplete captions. By understanding this link, researchers can better mitigate these biases through targeted data filtering and caption enrichment.

What Problem Does This Paper Address?

Contrastive Vision-Language Models (VLMs) are incredibly powerful for zero-shot tasks, but they have a glaring weakness: they excel at identifying objects (e.g., "cat") but struggle with attributes (e.g., "striped" or "yellow"). Prior literature has frequently blamed this on two distinct phenomena:

  1. The Modality Gap: A geometric quirk where image and text embeddings occupy completely separate regions of the shared vector space.
  2. Object Bias: The tendency of VLMs to prioritize object-level features over other contextual factors.

However, it has remained unclear why these phenomena emerge in the first place, whether they actually harm downstream performance, and if they are bugs or inherent features of contrastive learning. This paper systematically investigates these questions to uncover the underlying mechanics connecting the two effects.

Figure 1: Illustration of information imbalance between images (top left) and captions (bottom left). This imbalance makes it even for an oracle image encoder virtually impossible to predict the content of a caption, leading to undesirable ef- fects in contrastive training, such as the modality gap and object bias (see Section 6).

Methods

The authors approach the problem through a mix of large-scale empirical analysis and tightly controlled interventions:

  • New Metrics: They introduce the Relative Modality Gap (RMG) to measure embedding separation more accurately than standard L2 distance-to-means (L2M), which fails to account for per-sample pairings. They also introduce Matching Object Attribute Distance (MOAD) (and its generalized form, BRACE) to formally quantify object vs. attribute bias.
  • Dimensionality Ablation: They analyze the geometry of the modality gap by progressively zeroing out embedding dimensions to isolate which ones drive the separation.
  • Controlled Synthetic Testing: They build a synthetic framework to systematically vary the amount of information present in text captions while keeping the visual data constant.
  • Real-world Caption Manipulation: They train models on real-world data while artificially truncating captions to simulate varying levels of information scarcity.

Datasets

  • Evaluation Benchmarks: MS COCO (retrieval), ImageNet (zero-shot classification), MIT-States, and UT-Zappos (fine-grained attribute/object classification).
  • Pre-trained Models: 98 off-the-shelf VLMs (CLIP, SigLIP, etc.) trained on massive datasets like LAION-400M, LAION-2B, WebLI, and DataComp.
  • Multi-modal Attributes and Digits (MAD): A new, fully-controllable synthetic dataset introduced by the authors. Based on Morpho-MNIST, it maps digits to objects and applies controlled morphing operations (thickness, swelling, fractures, scaling, color) as attributes to study information imbalance cleanly.
Examples from the MAD dataset

Key Insights and Ideas

  • Information Imbalance is the "Trigger": Images naturally contain vastly more information than a short sentence can capture. Because an image encoder cannot guess which details a human decided to write down, it faces massive uncertainty. To minimize contrastive loss, the model prioritizes the most predictable elements (objects) and pushes the modalities apart to maximize uniformity.
  • The Modality Gap is Driven by Just a Few Dimensions: Surprisingly, the entire geometric separation between text and images is driven by only a few embedding dimensions. In fact, just two dimensions are often enough to perfectly separate the two modalities.
  • Closing the Gap Post-Hoc is a Bad Idea: Simply shifting or removing the gap-driving dimensions post-hoc does not improve downstream performance. This is because the local neighborhood structures of the image and text spaces are fundamentally different; they don't map 1:1.
  • The Gap is a Feature, Not a Bug: The modality gap provides the model with the mathematical flexibility to manage logit entropy (uncertainty). Forcing the gap closed during training without fixing the underlying data imbalance actually hurts the model's performance.
  • Object Bias is Actually "Caption Presence Bias": Models aren't biased toward objects because object words appear more frequently overall in a dataset (in fact, attributes often have higher global frequencies). Instead, it's driven by conditional probability: when an object is in an image, humans almost always name it, whereas they only selectively name its attributes.

Key Results

  • Confounder Correction: While a naive look at off-the-shelf models suggests a larger modality gap correlates with better performance, controlling for dataset and model size reveals the opposite: a smaller modality gap actually yields better downstream results.
  • Information vs. Gap: When training on the synthetic MAD dataset, increasing the number of attributes in the caption (reducing information imbalance) drastically shrinks the modality gap and object bias while boosting downstream accuracy.
  • Real-World Validation: On CC12M, models trained on "full captions" exhibited a significantly smaller modality gap and better retrieval performance ($R@1$) compared to models trained on "half" or "quarter" captions.
  • Fine-Tuning on Dense Data: Fine-tuning CLIP and SigLIP on the highly-aligned Densely Captioned Images (DCI) dataset successfully reduced the modality gap (lowering both L2M and RMG metrics).

Limitations and Future Work

  • Compute Constraints: Due to resource limitations, the authors could not train models from scratch on massive, web-scale datasets like LAION; validation of the training hypotheses was restricted to small-to-medium datasets (MAD, CC3M, CC12M).
  • Crude Imbalance Simulation: Dropping contiguous chunks of text (halves/quarters) to simulate information imbalance on real data is a rough heuristic and may introduce high variance.
  • Future Directions: The authors suggest exploring techniques from partial multi-label learning to help VLMs handle missing caption information more gracefully without developing geometric gaps or concept biases.

Summary updated

Abstract

Contrastive vision-language models (VLMs), like CLIP, have gained popularity for their versatile applicability to various downstream tasks. Despite their successes in some tasks, like zero-shot object recognition, they perform surprisingly poor on other tasks, like attribute recognition. Previous work has attributed these challenges to the modality gap, a separation of image and text in the shared representation space, and to a bias towards objects over other factors, such as attributes. In this analysis paper, we investigate both phenomena thoroughly. We evaluated off-the-shelf VLMs and while the gap's influence on performance is typically overshadowed by other factors, we find indications that closing the gap indeed leads to improvements. Moreover, we find that, contrary to intuition, only few embedding dimensions drive the gap and that the embedding spaces are differently organized. To allow for a clean study of object bias, we introduce a definition and a corresponding measure of it. Equipped with this tool, we find that object bias does not lead to worse performance on other concepts, such as attributes per se. However, why do both phenomena, modality gap and object bias, emerge in the first place? To answer this fundamental question and uncover some of the inner workings of contrastive VLMs, we conducted experiments that allowed us to control the amount of shared information between the modalities. These experiments revealed that the driving factor behind both the modality gap and the object bias, is an information imbalance between images and captions, and unveiled an intriguing connection between the modality gap and entropy of the logits.
BibTeX
@inproceedings{schrodi2025two,
  author = {Simon Schrodi and David T. Hoffmann and Max Argus and Volker Fischer and Thomas Brox},
  title = {{Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=uAFHCZRmXk},
}
Figure