Overcoming the Bag of Words Bias in Multimodal Models

41 concepts40 papers2 proposals

Contrastive vision-language models (VLMs) like CLIP have revolutionized zero-shot image classification and text-to-image retrieval by mapping images and text into a shared embedding space. However, these models suffer from a glaring structural blind spot: they frequently treat sentences as an unordered "bag of words."

If a model is presented with the caption "some plants surrounding a lightbulb" versus "a lightbulb surrounding some plants", it often assigns nearly identical similarity scores to both, completely ignoring grammar and word order. This phenomenon was systematically exposed by Tristan Thrush et al. in the Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality benchmark, and further formalized by Mert Yüksekgönül et al. through the ARO benchmark in When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?.

To understand why this happens, we have to look at how these models are trained. Standard contrastive pretraining uses random batches where negative examples are completely distinct images (e.g., matching a picture of a dog with a caption about a car). To maximize the contrastive objective, the model only needs to detect the presence of key entities—nouns like "dog" or "car"—rather than their structural relationships or attribute bindings. It takes the statistical path of least resistance: keyword matching.

Where Does the Structure Break?

To fix this bias, we must first locate where the structural information is lost. Does the text encoder fail to parse the grammar, or does the image encoder fail to see the relations?

Jin Wang et al. investigated this from a game-theoretic perspective in Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View, revealing a surprising asymmetry: while the text encoder often successfully preserves syntactic nuances internally, the image encoder and the final cross-modal alignment step fail to utilize them.

This insight is taken further by Darina Koishigarina et al. in CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modally. The authors demonstrate that individual uni-modal embeddings actually do encode relational structure (e.g., distinguishing a red cube from a blue sphere). However, because the cross-modal projection layer is trained on coarse, web-scale image-text pairs, it acts as a low-pass filter that discards these fine-grained structural dependencies during alignment. The information is present uni-modally, but it is crushed during cross-modal projection.

Forcing Structure via Hard Negatives

The most prominent strategy to fix this behavior is modifying the contrastive learning objective using hard negatives. If a model is forced during fine-tuning to distinguish between "a dog chasing a cat" and "a cat chasing a dog," it can no longer rely on simple keyword presence.

In When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?, the authors proposed NegCLIP, which generates hard negative text descriptions by shuffling word order or swapping attributes within the batch. Similarly, Maitreya Patel et al. introduced TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives, leveraging large language models (LLMs) and text-to-image diffusion models to synthesize paired text-and-image hard negatives. By training on these tightly coupled counterfactuals, the model learns to bind attributes to the correct nouns.

To prevent the model from losing its general zero-shot capabilities while learning structure, frameworks like CE-CLIP by Le Zhang et al. introduce adaptive ranking losses in Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding. This ensures that the model balances standard cross-modal alignment with fine-grained intra-modal discrimination.

Explicit Structural Priors and Graphs

Instead of relying on random text permutations, another line of work explicitly injects linguistic and visual structure into the training process using graph representations.

Yufeng Huang et al. proposed Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations, which utilizes Scene Graph Knowledge (SGK) to construct semantic negatives. By explicitly mapping objects, attributes, and relations into a graph structure, the model learns to encode the exact roles entities play in a scene.

Taking a causal approach, Fiorenzo Parascandolo et al. introduced Causally-Ordered Generative Training (COGT) in Causal Graphical Models for Vision-Language Compositional Understanding. By using dependency parsing to build a causal graphical model of descriptions, they force the network to process sentences sequentially based on structural hierarchy rather than treating them as unordered sets.

In a similar vein, Ziyi Shou et al. utilize Abstract Meaning Representation (AMR) graphs in Enhancing Semantic Understanding in Vision Language Models Using Meaning Representation Negative Generation to generate fluent, structurally permuted negative examples that target specific semantic roles, ensuring the model learns "who did what to whom."

Post-Hoc and Inference-Time Corrections

Can we fix the bag-of-words bias without completely retraining massive foundation models? Recent work suggests two non-destructive paths: test-time logical verification and post-hoc geometric transformations.

At inference time, Paola Cascante-Bonilla et al. proposed Caption Expansion with Contradictions and Entailments (CECE) in Natural Language Inference Improves Compositionality in Vision-Language Models. Instead of relying on a single cross-modal score, an LLM generates logical entailments and contradictions for the target caption. The vision-language model then evaluates this expanded set of statements, using natural language inference as a "double-check" mechanism to resolve structural ambiguities.

Alternatively, if the structural information is already present but suppressed in the joint embedding space, we can fix it geometrically. As shown by Darina Koishigarina et al. with LABCLIP, learning a lightweight, post-hoc linear transformation on top of frozen CLIP embeddings can successfully recover the hidden compositional structure, bypassing the need for expensive fine-tuning entirely.

Key Foundational Work

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (2022) — contributed 1 concept elements to this direction.
Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies? (2022) — contributed 1 concept elements to this direction.
Bridge To Answer: Structure-Aware Graph Interaction Network for Video Question Answering (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bag_of_words_bias_fd2fbfd9,
  title = {bag-of-words bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bag-of-words-bias/rdr_fd2fbfd931e441a788231c8976dd4fc5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bag_of_words_bias_fd2fbfd9_v1,
  title = {bag-of-words bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bag-of-words-bias/rdr_fd2fbfd931e441a788231c8976dd4fc5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake