When a human learns what a "cube" is and what "transparent" means, they instantly understand the concept of a "transparent cube," even if they have never seen one. This ability to flexibly combine known primitives into novel configurations is called compositional generalization.
State-of-the-art Vision-Language Models (VLMs) like CLIP or GPT-4V appear to possess this capability on standard benchmarks. However, closer inspection reveals a profound compositional reasoning gap. While these models excel at recognizing individual objects or attributes in isolation, they frequently fail when those same concepts are rearranged into unfamiliar combinations.
Understanding why this gap exists—and how researchers are trying to close it—requires looking under the hood at how VLMs encode language and vision.
The Bag-of-Words Illusion
To understand why compositionality fails, we must first understand how VLMs "read" scenes. In text-image matching paradigms, models are trained to align visual representations with textual descriptions. Ideally, the model should parse the syntax of a sentence to understand the relationships between words.
In practice, VLMs often take a statistical shortcut. As Yuksekgonul et al. demonstrate, models frequently behave like a "bag of words." They struggle to distinguish between sentences like "the horse is eating the grass" and "the grass is eating the horse." Because both sentences contain identical tokens, their aggregated text embeddings land in nearly the same spot in the shared embedding space.
This limitation extends directly to attribute binding. A single-author paper by Gaur introduces the ColorSwap benchmark to test whether models can bind colors to the correct objects. When presented with an image of a white cat and a black vase, top-tier models routinely confuse it with an image of a black cat and a white vase. The models recognize the presence of "cat," "vase," "black," and "white," but they fail to ground the linguistic structure that dictates which color belongs to which object.
Why Models Cheat: Phrase Memorization
Why do massive neural networks fall back on a bag-of-words strategy? The root cause lies in a fundamental training bias: phrase memorization.
During pre-training on billions of web images, models encounter common word co-occurrences. If a model sees thousands of examples of "white cats" and "brown dogs," it optimizes its weights to recognize these holistic visual patterns. Instead of learning "white" and "cat" as independent, composable primitives, it memorizes the phrase as a single, monolithic unit (Cheng et al.; Nikolaus et al.).
Holistic Pattern Matching (Standard VLM):
["white cat"] ───> [Single Visual Representation]
["white dog"] ───> ??? (Fails to generalize out-of-distribution)
Disentangled Representation (Ideal Compositional VLM):
["white"] ───> [Color Feature] ┐
├─> [Combined Representation]
["dog"] ───> [Object Feature]┘
When tested on a novel combination like "white dog," the model experiences a systematicity failure. As Jin et al. observe in continual learning settings, models exhibit a massive performance drop between seen and unseen word combinations. Rather than grounding individual concepts for flexible recombination, they rely on spurious correlations within the training distribution (Kafle et al.).
This lack of true compositionality has been verified by probing the internal representations of foundation models. Yun et al. developed a framework called CompMap to test whether models use primitive concepts (like color or shape) to identify complex objects. They discovered that high zero-shot accuracy is driven by holistic pattern matching rather than compositional reasoning. Rohban et al. formalize this by showing that a model's ability to generalize to unseen attribute-object pairings (such as a "chocolate beaver") is directly tied to its level of representation disentanglement—the degree to which attributes and objects are encoded as independent factors.
Crucially, this is not a problem that disappears with scale. The CREPE benchmark, introduced by Ma et al., reveals that scaling up data volume and parameter size does not inherently fix compositionality failures. Even frontier models like GPT-4o struggle with fine-grained reasoning on complex, multi-concept descriptions, lagging far behind human-level proficiency (Hua et al.).
Strategies for Building Compositional Intuition
To bridge the compositional reasoning gap, researchers are shifting away from brute-force scaling and toward architectural and training mechanisms that force models to learn structure.
1. Hard Negative Contrastive Learning
If standard contrastive learning allows models to cheat by using a bag-of-words strategy, the solution is to make the training data harder. Yuksekgonul et al. proposed NegCLIP, an approach that augments training with "hard negatives"—text descriptions where the words are syntactically permuted (e.g., swapping subject and object). By forcing the model to maximize the similarity score for the correct text while minimizing it for the permuted text, the model is structurally incentivized to learn word order and relational logic.
2. Meta-Learning and Virtual Loops
To break the habit of phrase memorization, researchers use meta-learning frameworks to simulate the experience of encountering novel concepts. For instance, the MCRES framework developed by Xu et al. uses virtual training and testing loops during referring expression segmentation. By intentionally withholding certain attribute-object combinations in the virtual training phase and testing on them in the virtual test phase, the model is penalized if it attempts to memorize phrases as single units. Similarly, Xu et al. introduced MetaReVision, which combines retrieval-augmented learning with meta-learning to allow models to fast-adapt to novel concept combinations at test time by retrieving relevant isolated primitives.
3. Structural Decomposition and In-Context Guidance
Instead of forcing a single neural network to process a complex scene all at once, another direction breaks the problem down into explicit, logical steps.
- Question Decomposition: Khan et al. introduced a selective decomposition strategy for Visual Question Answering (VQA). When a model is uncertain about a complex compositional question, it dynamically breaks it down into simpler sub-questions (e.g., verifying the attribute and the object independently), preventing the model from relying on surface-level word statistics.
- Structural In-Context Learning: For Large Vision-Language Models (LVLMs), Li et al. demonstrated that selecting in-context examples based on structural similarity—while filtering out distracting visual noise—allows models to generalize to novel compositions without any parameter updates.
- Neuro-Symbolic Pipelines: Early foundational work like Neural Baby Talk (Lu et al.) bypassed holistic generation entirely by creating sentence templates with explicit slots tied to object detectors, ensuring that language generation remains strictly grounded in visual regions.
4. Repurposing Language Models for Spatial Reasoning
Because Large Language Models (LLMs) are trained on massive text corpora, they possess a highly robust, inherent understanding of compositional grammar and spatial logic. Rather than training a vision model from scratch to learn compositionality, Kulits et al. developed IG-LLM, which repurposes an LLM to solve the "inverse graphics" problem (converting 2D images into 3D renderable code). By adding a specialized numeric head to the language model, they leverage the LLM's internal structural representations, allowing the system to maintain high accuracy on out-of-distribution, novel attribute-shape combinations where traditional modular vision pipelines fail.
Summary
The compositional reasoning gap highlights a fundamental difference between statistical pattern matching and true understanding. Current VLMs often succeed by recognizing the presence of elements rather than their organization. By implementing hard negative training, meta-learning loops, and explicit structural decomposition, the field is moving toward models that do not just memorize the world, but systematically compose it.
Key Foundational Work
Neural Baby Talk (2018) — contributed 1 concept elements to this direction.
Language Models for Image Captioning: The Quirks and What Works (2015) — contributed 1 concept elements to this direction.