The Compositional Reasoning Gap in Vision Language Models

101 concepts84 papers2 proposals

State-of-the-art vision-language models (VLMs) like CLIP are remarkably adept at identifying objects, textures, and colors within an image. However, if you present them with an image of a "man biting a dog" alongside the caption "a dog biting a man," these models frequently fail to see the difference.

This phenomenon is known as the compositional reasoning gap. Despite their massive scale, contrastive vision-language models often operate as sophisticated "bags of words." They excel at entity recognition but struggle to comprehend the syntax, spatial relationships, and structural dependencies that bind these entities together.

Understanding why this gap exists—and how researchers are attempting to close it—requires looking closely at how these models encode information.

The Root of the Problem: Global Compression and Textual Shortcuts

To understand why VLMs lose track of structure, we have to look at their training objective. Standard contrastive learning forces an entire image and an entire caption into single, low-dimensional vector embeddings, optimizing for the global similarity between the two.

When a model compresses a complex visual scene into a single vector, fine-grained spatial arrangements are often the first things to be discarded. If an image contains a "cube" and a "sphere," and the colors "red" and "blue," a contrastive model can achieve a near-perfect alignment score simply by verifying the presence of these four individual concepts. It does not need to learn the explicit binding—whether the cube is red or the sphere is blue—to satisfy its training objective.

As a result, models learn to "cheat." As shown by Li et al., VLMs frequently rely on textual biases and prior probabilities within the training data rather than truly grounding the text in the image structure. If the text sounds plausible, the model scores it highly, regardless of whether the spatial layout matches.

Probing the Failure Modes

To systematically expose this vulnerability, Thrush introduced the Winoground benchmark. Winoground uses "minimal pairs": two images and two captions containing the exact same words, but arranged differently (e.g., "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants"). When forced to match the correct image to the correct caption, even advanced models often perform no better than random chance.

Similarly, Yuksekgonul et al. developed the ARO benchmark to test sensitivity to attribute binding and relation ordering. Their findings confirmed that standard pre-training paradigms fail to penalize models for ignoring word order, cementing the "bag-of-words" behavior.

Standard VLM Pipeline (Bag-of-Words Vulnerability):
Image ---> [Visual Encoder] ---> Global Vector \
                                                 ---> High Similarity (Fails to bind relations)
Text  ---> [Text Encoder]   ---> Global Vector /
("Grass eating a horse")

Mechanism 1: Hard Negative and Positive Fine-Tuning

The most direct way to force a model to pay attention to structure is to alter its training data. If a model is only exposed to correct image-text pairs and random, completely different negatives (like an image of a dog paired with the text "a car"), it never learns to appreciate syntax.

Yuksekgonul et al. proposed NegCLIP, which augments contrastive training with "hard negatives"—captions where the nouns or adjectives are deliberately swapped. By forcing the model to distinguish between "a horse eating grass" and "grass eating a horse," the model is penalized unless it learns to encode the structural relationship between the subject and the object.

However, fixing compositionality is rarely that simple. Kamath et al. discovered that models trained exclusively on hard negatives become oversensitive "perturbation detectors." They begin to reject valid sentences simply because a word was changed, failing to recognize synonymous "hard positives." True compositional grounding requires balancing both structural negatives and semantic variations during training.

Mechanism 2: Generative Models as Zero-Shot Classifiers

A fascinating alternative to contrastive models comes from the world of image generation. Li et al. demonstrated that pre-trained text-to-image diffusion models can be inverted to act as highly accurate zero-shot classifiers.

Why do diffusion models possess a superior grasp of compositionality compared to CLIP? The answer lies in their architecture. To generate an image from the prompt "a red cube on top of a blue sphere," a diffusion model cannot simply rely on a global vector. At every step of the denoising process, its internal cross-attention mechanisms must physically map the textual tokens ("red", "cube", "on top of") to specific spatial coordinates and pixel patches.

When we use a diffusion model for classification—by calculating which text prompt best guides the denoising of a given image—we are leveraging this pixel-level relational grounding. As noted by Uselis et al., while diffusion classifiers face challenges like domain gaps between real and generated styles, their underlying cross-attention maps inherently preserve spatial and relational dependencies that contrastive models discard.

Mechanism 3: Dense Tokenization and Explicit Structure

Another prominent direction moves away from compressing an image into a single global vector, opting instead to preserve spatial granularity.

Large Multimodal Models (LMMs) like LLaVA perform significantly better at compositional tasks than standard CLIP, even when they use the exact same frozen vision encoder weights. Koh et al. investigated this discrepancy and found that the secret lies in how the visual data is fed to the language model. Instead of pooling the vision features into a single embedding, LMMs pass dense patch tokens directly to the LLM. The autoregressive attention layers of the language model can then dynamically "reweight" and bind these visual patches to specific words in the prompt, reconstructing spatial orientation on the fly.

Taking this structural preservation a step further, Mitra et al. introduced Compositional Chain-of-Thought (CCoT). Instead of forcing the model to map an image directly to an answer, CCoT prompts the model to first generate an intermediate Scene Graph—explicitly parsing the objects, their attributes, and their spatial relationships. By breaking the visual reasoning down into an explicit, structured symbolic step, the model avoids the bag-of-objects trap entirely.

Compositional Chain-of-Thought (CCoT) Pipeline:
Image ---> Dense Patches ---> [Scene Graph Generation] ---> [Final Reasoning]
                               (Object: Cube, Color: Red)
                               (Object: Sphere, Color: Blue)
                               (Relation: Cube ON TOP OF Sphere)

By shifting from coarse global alignment to granular, attention-guided, and structurally explicit representations, current research is steadily transforming vision-language models from simple pattern matchers into systems capable of true visual syntax.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compositional_reasoning_failure_composit_a939afaa,
  title = {compositional reasoning failure (compositional reasoning gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-reasoning-failure-compositional-reasoning-gap/rdr_a939afaac0d34e2fb75bd5733bdbcda9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_compositional_reasoning_failure_composit_a939afaa_v1,
  title = {compositional reasoning failure (compositional reasoning gap)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compositional-reasoning-failure-compositional-reasoning-gap/rdr_a939afaac0d34e2fb75bd5733bdbcda9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake