Modern vision-language models (VLMs) like CLIP are remarkably adept at identifying objects, yet they frequently fail to understand how those objects relate to one another. This phenomenon is often described as "bag-of-words" behavior: the model recognizes the presence of "horse" and "grass" but cannot distinguish between "the horse is eating the grass" and "the grass is eating the horse."
Research across hundreds of papers suggests that while our models have mastered visual vocabulary, they remain "compositionally illiterate," struggling to synthesize individual entities into structured, relational scenes.
The Compositionality Gap
The core issue is a discrepancy between entity recognition and relational understanding. As Mert Yuksekgonul and Federico Bianchi demonstrate in When and why vision-language models behave like bags-of-words, and what to do about it?, models often rely on the mere presence of keywords rather than their syntactic arrangement.
This failure is most visible in "winograd-style" tests. In the Winoground benchmark, Tristan Thrush and colleagues show that state-of-the-art models often perform near random chance when asked to match images to captions where the words are identical but the meaning is swapped (e.g., "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants").
Why Global Embeddings Collapse Structure
The "bag-of-words" behavior isn't just a training deficiency; it may be a geometric limitation of how we represent information. Standard VLMs compress an entire image or sentence into a single global vector. Raphi Kang and Georgia Gkioxari argue in Is CLIP ideal? No. Can we fix it? Yes! that this single-vector latent space is mathematically incapable of simultaneously handling attribute binding, spatial relations, and negation. When you force a complex scene into one point in space, the fine-grained "topology" of how objects interact is lost.
This leads to several specific failure modes:
- Attribute Binding: Failing to link a color or size to the correct object (e.g., "a red cube and a blue sphere" vs. "a blue cube and a red sphere").
- Spatial Relations: Confusion over "left of," "inside," or "underneath," as noted by James Seale Smith in the ConStruct-VL framework.
- Action Directionality: Identifying the verb but not who is doing it to whom, a problem highlighted by Santiago Castro in Scalable Performance Analysis for Vision-Language Models.
Teaching Models to See Structure
To move beyond the bag-of-words stage, researchers are exploring three primary strategies:
1. Hard Negative Contrastive Learning
If a model thinks "dog bites man" and "man bites dog" are the same, we can explicitly train it to see the difference. Yuksekgonul et al. proposed NegCLIP, which uses "composition-aware" negatives—captions where nouns or relations are shuffled—to force the model to pay attention to structure. Similarly, Dylan Sam introduced PC-CLIP (Finetuning CLIP to Reason about Pairwise Differences), which aligns the vector differences between image embeddings with the textual descriptions of those differences.
2. Neurosymbolic and Graph Representations
Instead of a flat vector, some researchers use Scene Graphs to explicitly model entities as nodes and relationships as edges. Yu-Feng Huang and Zhuo Chen developed Structure-CLIP, which integrates scene graph knowledge to prevent the collapse of structural information. In the generative space, Frank Fundel demonstrated that conditioning diffusion models on scene graphs allows for far more precise spatial control than text alone.
3. Compositional Reasoning Paths
Rather than changing the architecture, we can change how the model "thinks" at inference time. Chancharik Mitra and Trevor Darrell introduced Compositional Chain-of-Thought (CCoT) (Compositional Chain-of-Thought Prompting for Large Multimodal Models), which prompts models to first generate a structured scene graph of an image before attempting to answer complex questions about it. This intermediate step forces the model to ground its reasoning in the specific relational dependencies of the scene.
Communicative Decoding
A more dynamic approach is Communicative Decoding, proposed by Delin Chen and Chuang Gan in CoVLM. Instead of a one-way pass from vision to language, the model engages in a dialogue. It uses specialized tokens to "ask" the vision system to ground specific nouns and verbs into image regions during the generation process, ensuring that the relationships it describes are actually present in the visual data.
By moving from global, unordered embeddings to structured, relational representations, the field is beginning to bridge the gap between simple object detection and true visio-linguistic understanding.