Abstract: Contrastive vision-language models like CLIP are widely used for zero-shot classification and as vision encoders for multi-modal systems. Despite their popularity, their representations show major limitations in understanding structural relationships. Specifically, CLIP models learn "bag-of-words" representations, failing to distinguish between an image of "a yellow submarine and a blue bus" and "a blue submarine and a yellow bus." While previous attempts focused on architectural changes or hard negative mining, this work identifies the training data as the primary culprit. Using a synthetic dataset (MADMAN), we demonstrate that common properties of natural data—such as saliency bias, low attribute density, and incomplete captions—discourage models from learning object-attribute binding. We find that neither scaling batch sizes nor explicit hard negatives solve the problem; only by optimizing the data properties themselves can CLIP achieve reliable binding.
The Problem: Bag-of-Words Representations
Contrastive Vision-Language models like CLIP are the backbone of modern AI, powering everything from DALL-E to advanced zero-shot classifiers. However, they suffer from a fundamental "Bag-of-Words" (BOW) problem. They are excellent at identifying that an image contains specific concepts like "yellow," "blue," "submarine," and "bus," but they struggle with object-attribute binding—the ability to correctly associate which color belongs to which vehicle.
Previous research attempted to fix this by modifying model architectures or adding "hard negatives" (incorrectly bound captions) during training. These attempts yielded only marginal improvements. This research asks a fundamental question: Is the problem hidden in the training data? Specifically, do the messy, incomplete, and biased captions found on the web actually discourage models from learning how to bind attributes to objects?
Methodology: The MADMAN Dataset
To test this hypothesis, the authors developed MADMAN (Multi-Attribute and Digit for Multi-Attribute biNding), a fully controllable synthetic dataset. MADMAN uses MNIST digits as objects and applies various transformations—such as color, rotation, thickness, and swelling—as attributes.
Because the data is synthetic, researchers could perform "data interventions," precisely varying one property at a time (like the number of objects or the frequency of specific attributes) while keeping others constant. They compared a "Realistic" setup, tuned to match the statistics of the CC12M (Conceptual Captions 12M) dataset, against an "Ideal" setup to isolate the drivers of learning.
Key Insights: Why Binding Fails
The researchers identified four specific data properties that act as "binding killers" in natural datasets:
1. The Saliency Shortcut
Humans tend to describe the most "salient" (prominent) object first. If a model can minimize its loss just by looking at the center or most prominent object and ignoring the rest, it will never learn the harder task of binding attributes for multiple objects in a scene.
2. The "Goldilocks" Zone of Attributes
There is an inverse U-shaped relationship between the number of attributes in a caption and binding accuracy.
- Too few attributes: The model doesn't receive enough signal to learn the relationship.
- Too many attributes: A "combinatorial explosion" occurs where the unique set of words is so specific that the model can identify the image just by the "bag of words" without needing to understand the structure.
3. Caption Incompleteness
In real-world web data, captions rarely describe every object. If most training data only describes one of two objects, the model learns that a partial bag-of-words is sufficient for a match, bypassing the need for structural understanding.
4. Data Trumps Scale
Surprisingly, neither increasing the batch size (which implicitly adds more negatives) nor using larger models fixed the binding issue on "realistic" data. Only changing the properties of the data itself led to near-perfect binding performance.
Key Results and Conclusion
The experiments yielded stark contrasts between data environments:
- The "Ideal" Boost: When trained on "Ideal" data (multiple objects, no saliency bias, and balanced attribute density), CLIP achieved 90-100% binding accuracy across most attributes.
- Realistic Failure: When the synthetic data was tuned to match real-world "Realistic" parameters, accuracy plummeted to near-chance levels (~50-55%).
- Hard Negatives Are Insufficient: Testing NegCLIP (a version of CLIP trained specifically with hard negatives) showed that while it improved performance slightly, it could not match a standard CLIP model trained on "Ideal" data.
This research proves that data quality and composition are more impactful than loss-function tweaks. To solve the binding problem, future work should focus on automated filtering for "high-binding-potential" samples, using LLMs to re-caption images to be more complete and less biased, or implementing training techniques like "salient dropping" to force models to attend to less prominent objects.
Summary updated