Learning the Power of "No": Foundation Models with Negations

Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, Aparna BharatiWACV 2025 · DOI

Introduction

Current Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable capabilities in zero-shot image classification and retrieval. However, these models often exhibit a "bag-of-words" behavior, where they excel at identifying individual objects but fail to grasp the relational semantics that bind them. A particularly glaring weakness is the inability to process negations. For instance, a prompt like "a dog without ears" often yields the same result as "a dog with ears," as the model prioritizes the presence of the keyword "dog" over the logical modifier "without."

Comparison of state-of-the-art generative models failing to respect negation prompts.

Addressing the Negation Gap

To bridge this gap, researchers introduced CC-Neg, a large-scale dataset of over 228,000 image-caption pairs specifically curated to teach models the semantics of negation. Alongside this dataset, the CoN-CLIP (Contrastive Negation CLIP) framework was developed to fine-tune existing models.

Methodology

The approach involves two primary innovations:

  1. Automated Negation Generation: Leveraging the PaLM-2 Large Language Model, the team systematically negated captions from the Conceptual Captions (CC-3M) dataset. This process transformed positive descriptions (e.g., "A car with tires") into fluent negations (e.g., "A car without tires").
  2. Contrastive Negation Loss: CoN-CLIP modifies the standard CLIP training objective by adding "repelling" forces. It pushes image representations away from their negated textual counterparts and utilizes "distractor images" to further refine the model's understanding of what an attribute's absence looks like.

Key Findings and Performance

The study reveals that teaching a model negation serves as a powerful proxy for broader compositional understanding. By learning what "no tires" means, the model is forced to more precisely define the visual features and spatial context of "tires."

Results

  • Negation Accuracy: On the CC-Neg benchmark, CoN-CLIP achieved a near-perfect 99.7% accuracy in distinguishing true captions from negated ones, a massive leap from the base CLIP model's ~66%.
  • Zero-Shot Generalization: The negation-aware training improved standard zero-shot classification by an average of 3.85% across eight datasets, with a notable 10.95% jump on CIFAR-100.
  • Compositional Reasoning: The model outperformed base CLIP by 4.4% on the SugarCREPE benchmark, suggesting that understanding "not" enhances the model's grasp of object arrangements and attributes.

Limitations and Future Directions

Despite these gains, challenges remain. Performance still scales inversely with the complexity of the scene; as the number of objects and relations increases, the model's logical reasoning begins to falter. Furthermore, the "distractor images" used in training are currently selected via keyword matching rather than precise synthesis. Future research aims to integrate negation-aware objectives directly into the initial pre-training phase of foundation models to foster more robust "common sense" reasoning from the ground up.

Summary updated

BibTeX
@inproceedings{singh2025learning,
  author = {Jaisidh Singh and Ishaan Shrivastava and Mayank Vatsa and Richa Singh and Aparna Bharati},
  title = {{Learning the Power of "No": Foundation Models with Negations}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00777},
}
Figure