CLIP Won't Learn Object-Attribute Binding from Natural Data and Here is Why

Bijay Gurung, David T. Hoffmann, Thomas BroxarXiv 2025 · OpenReview

Abstract: Contrastive vision-language models like CLIP are widely used for zero-shot classification and as vision encoders for multi-modal systems. Despite their popularity, their representations show major limitations in understanding structural relationships. Specifically, CLIP models learn "bag-of-words" representations, failing to distinguish between an image of "a yellow submarine and a blue bus" and "a blue submarine and a yellow bus." While previous attempts focused on architectural changes or hard negative mining, this work identifies the training data as the primary culprit. Using a synthetic dataset (MADMAN), we demonstrate that common properties of natural data—such as saliency bias, low attribute density, and incomplete captions—discourage models from learning object-attribute binding. We find that neither scaling batch sizes nor explicit hard negatives solve the problem; only by optimizing the data properties themselves can CLIP achieve reliable binding.

The Problem: Bag-of-Words Representations

Contrastive Vision-Language models like CLIP are the backbone of modern AI, powering everything from DALL-E to advanced zero-shot classifiers. However, they suffer from a fundamental "Bag-of-Words" (BOW) problem. They are excellent at identifying that an image contains specific concepts like "yellow," "blue," "submarine," and "bus," but they struggle with object-attribute binding—the ability to correctly associate which color belongs to which vehicle.

CLIP fails to distinguish between shuffled captions, assigning identical similarity scores to correct and incorrect attribute bindings.

Previous research attempted to fix this by modifying model architectures or adding "hard negatives" (incorrectly bound captions) during training. These attempts yielded only marginal improvements. This research asks a fundamental question: Is the problem hidden in the training data? Specifically, do the messy, incomplete, and biased captions found on the web actually discourage models from learning how to bind attributes to objects?

Methodology: The MADMAN Dataset

To test this hypothesis, the authors developed MADMAN (Multi-Attribute and Digit for Multi-Attribute biNding), a fully controllable synthetic dataset. MADMAN uses MNIST digits as objects and applies various transformations—such as color, rotation, thickness, and swelling—as attributes.

Because the data is synthetic, researchers could perform "data interventions," precisely varying one property at a time (like the number of objects or the frequency of specific attributes) while keeping others constant. They compared a "Realistic" setup, tuned to match the statistics of the CC12M (Conceptual Captions 12M) dataset, against an "Ideal" setup to isolate the drivers of learning.

Key Insights: Why Binding Fails

The researchers identified four specific data properties that act as "binding killers" in natural datasets:

1. The Saliency Shortcut

Humans tend to describe the most "salient" (prominent) object first. If a model can minimize its loss just by looking at the center or most prominent object and ignoring the rest, it will never learn the harder task of binding attributes for multiple objects in a scene.

2. The "Goldilocks" Zone of Attributes

There is an inverse U-shaped relationship between the number of attributes in a caption and binding accuracy.

  • Too few attributes: The model doesn't receive enough signal to learn the relationship.
  • Too many attributes: A "combinatorial explosion" occurs where the unique set of words is so specific that the model can identify the image just by the "bag of words" without needing to understand the structure.

3. Caption Incompleteness

In real-world web data, captions rarely describe every object. If most training data only describes one of two objects, the model learns that a partial bag-of-words is sufficient for a match, bypassing the need for structural understanding.

Comparison between a setup where Bag-of-Words is sufficient (b) and an improved setup where binding is required (c) to distinguish between images.

4. Data Trumps Scale

Surprisingly, neither increasing the batch size (which implicitly adds more negatives) nor using larger models fixed the binding issue on "realistic" data. Only changing the properties of the data itself led to near-perfect binding performance.

Key Results and Conclusion

The experiments yielded stark contrasts between data environments:

  • The "Ideal" Boost: When trained on "Ideal" data (multiple objects, no saliency bias, and balanced attribute density), CLIP achieved 90-100% binding accuracy across most attributes.
  • Realistic Failure: When the synthetic data was tuned to match real-world "Realistic" parameters, accuracy plummeted to near-chance levels (~50-55%).
  • Hard Negatives Are Insufficient: Testing NegCLIP (a version of CLIP trained specifically with hard negatives) showed that while it improved performance slightly, it could not match a standard CLIP model trained on "Ideal" data.

This research proves that data quality and composition are more impactful than loss-function tweaks. To solve the binding problem, future work should focus on automated filtering for "high-binding-potential" samples, using LLMs to re-caption images to be more complete and less biased, or implementing training techniques like "salient dropping" to force models to attend to less prominent objects.

Summary updated

Abstract

Contrastive vision-language models like CLIP are used for a large variety of applications, such as zero-shot classification or as vision encoder for multi-modal models. Despite their popularity, their representations show major limitations. For instance, CLIP models learn bag-of-words representations and, as a consequence, fail to distinguish whether an image is of ``a yellow submarine and a blue bus'' or ``a blue submarine and a yellow bus''. Previous attempts to fix this issue added hard negatives during training or modified the architecture, but failed to resolve the problem in its entirety. We suspect that the missing insights to solve the binding problem for CLIP are hidden in arguably the most important part of learning algorithms: the data. In this work, we fill this gap by rigorously identifying the influence of data properties on CLIP's ability to learn binding using a synthetic dataset. We find that common properties of natural data such as low attribute density, incomplete captions, and the saliency bias, a tendency of human captioners to describe the object that is ``most salient'' to them, have a detrimental effect on binding performance. In contrast to common belief, we find that neither scaling the batch size, i.e., implicitly adding more hard negatives, nor explicitly creating hard negatives enables CLIP to learn reliable binding. Only when the data expresses our identified data properties does CLIP learn almost perfect binding.
BibTeX
@misc{gurung2025clip,
  author = {Bijay Gurung and David T. Hoffmann and Thomas Brox},
  title = {{CLIP Won't Learn Object-Attribute Binding from Natural Data and Here is Why}},
  year = {2025},
  url = {https://openreview.net/forum?id=VZMYUHV1xj},
}
Figure