Overcoming the Bag-of-Words Bias in Vision-Language Models

This direction is split into other directions: bag-of-words bias. Its last version stays readable and citable here.

Version 1 of 184 papers92 conceptsfrozen 2026-10-04

Large-scale Vision-Language Models (VLMs) like CLIP have revolutionized multimodal AI, yet they harbor a surprising "blind spot." While they excel at identifying individual objects, they often fail to understand how those objects relate to one another. If you show a standard VLM an image of "a dog chasing a cat" and a caption saying "a cat chasing a dog," the model will likely give them a high similarity score.

This phenomenon is known as the bag-of-words bias. The model treats a caption not as a structured sentence, but as an unordered collection of keywords. As long as the words "dog," "cat," and "chasing" are present, the model is satisfied. Recent research has focused on moving beyond this lexical matching toward true compositional reasoning—the ability to understand how attributes, relations, and word order combine to form specific meanings.

The Compositional Reasoning Gap

The depth of this problem was famously highlighted by Tristan Thrush and colleagues in their work on Winoground. They introduced a benchmark of "minimal pairs": two images and two captions that share the exact same vocabulary but differ in meaning (e.g., "some plants surrounding a lightbulb" vs. "a lightbulb surrounding some plants"). They found that state-of-the-art models often perform no better than random chance on these tasks.

The intuition behind this failure lies in the training objective. Standard contrastive learning encourages the model to maximize the similarity between an image and its global caption. As Assaf Arbelle et al. argue in Teaching Structured Vision&Language Concepts to Vision&Language Models, models find "shortcuts" during training. Identifying a "dog" and a "rug" is usually enough to lower the loss, so the model never learns the subtle difference between "a dog on a rug" and "a rug on a dog."

Generative Negative Mining

One of the most effective ways to break this bias is to force the model to confront its mistakes during training. This is achieved through hard negative mining. If the model is only ever shown "correct" pairs and "randomly different" pairs (like a dog image vs. a car caption), it never learns fine-grained distinctions.

Researchers like Uğur Şahin in Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining and Maitreya Patel in TripletCLIP use a generative pipeline to create "foils." They use Large Language Models (LLMs) to swap subjects and objects in a caption and Diffusion models to generate synthetic images that match these swapped captions. By training the VLM to distinguish the original image from these "hard negatives," the model is forced to pay attention to structural details it previously ignored.

Injecting Structural Priors

Beyond just providing better data, some researchers argue that the model's internal representation must be explicitly structured. Instead of a single vector representing a whole sentence, these methods decompose the input into its constituent parts.

  • Scene Graphs: Yufeng Huang and the team behind Structure-CLIP integrate Scene Graph Knowledge (SGK). By converting captions into graphs of objects, attributes, and relations, they ensure the model encodes the specific "triples" (e.g., [Dog]—[On]—[Rug]) rather than just a list of words.
  • Causal Modeling: Fiorenzo Parascandolo introduced COGT (Causally-Ordered Generative Training), which uses dependency parsing to build a causal graphical model of a sentence. This guides the model to learn the directionality of relationships—knowing that the "chaser" and the "chased" occupy different roles in the causal chain.

Inference-Time Refinement

Interestingly, the knowledge required for compositional reasoning might already exist within pre-trained models, even if they don't use it by default. Darina Koishigarina et al. discovered in CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally that CLIP's individual encoders actually capture structural information, but this information is "washed out" during the final alignment step.

To solve this without expensive retraining, "training-free" methods like GroundingCLIP (GCLIP) by Madhukar Reddy Vongala and ComCLIP by Kenan Jiang decompose the task at inference time. They use an LLM to break a complex caption into sub-queries (e.g., "black cat," "white dog," "sitting next to") and then use object detectors to find these specific entities in the image. By checking if the image contains the right parts in the right configuration, they bypass the bag-of-words trap entirely.

Why It Matters

Moving from keyword matching to structural understanding is the difference between an AI that "sees" a scene and an AI that "understands" it. As Yan Huang notes in Investigating Compositional Challenges in Vision-Language Models for Visual Grounding, this capability is essential for high-stakes applications like robotics or autonomous driving, where the difference between "the car is in front of the truck" and "the truck is in front of the car" is a matter of safety, not just semantics.

Papers in this version

Showing 60 of 84 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overcoming_the_bag_of_words_bias_in_visi_dda34d55,
  title = {Overcoming the Bag-of-Words Bias in Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-bag-of-words-bias-in-vision-language-models/rdr_dda34d5562ea4fe182195d238f69a4a6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_overcoming_the_bag_of_words_bias_in_visi_dda34d55_v1,
  title = {Overcoming the Bag-of-Words Bias in Vision-Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overcoming-the-bag-of-words-bias-in-vision-language-models/rdr_dda34d5562ea4fe182195d238f69a4a6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}