Negation Blindness in Large Language Models: Unveiling the NO Syndrome in Image Generation

Mohammad Nadeem, Shahab Saquib Sohail, Erik Cambria, Björn W. Schuller, Amir HussainarXiv 2024 · DOI · arXiv

Introduction

Despite their multi-modal prowess, state-of-the-art Large Language Models (LLMs) suffer from "Negation Blindness"—a consistent failure to respect "NO" constraints in image generation prompts. This study identifies the "NO Syndrome," revealing that while models like GPT-4 and Gemini can verbally acknowledge a request to exclude an object, they almost invariably include it in the final visual output across multiple languages.

Trends in LLM and Generative AI research interest (2023-2024)

The Problem: Logical Gaps in Visual Synthesis

As LLMs evolve into versatile multimodal tools, researchers have cataloged flaws like hallucinations and social biases. However, the inability to process simple negations in visual contexts remains under-explored. If you ask a model to "generate an image of a person without spectacles," it often produces a portrait with glasses. This suggests a deep-seated architectural limitation in bridging logical linguistic constraints and visual synthesis.

Methodology and Datasets

The researchers evaluated GPT-4, Gemini, and Copilot using prompts in English, Hindi, and French. The study utilized a curated set of "Negation Prompts" targeting different exclusions:

  1. Q1: Person/dog with no spectacles.
  2. Q2: Elephant with no tusks.
  3. Q3: Flowers with no blue color.
  4. Q4: Market with no car.
  5. Q5: Car with no dog on top.

The methodology involved iterative testing (five runs per prompt) and statistical validation using the Friedman and Wilcoxon Signed-Rank Tests to assess consistency and semantic entropy.

Overview of the experimental methodology and visual examples of negation blindness

Key Insights: The Multimodal Disconnect

A striking finding is the "misalignment" between text and image. GPT-4 might state, "Here is an image of an elephant without tusks," while generating an image with massive tusks. This suggests that text and image generation modules are decoupled, lacking a feedback loop to verify if the visual output matches linguistic constraints.

Furthermore, the syndrome is language-invariant, suggesting the flaw lies in core conceptual processing. Models struggle most when asked to remove "essential" or high-frequency attributes (like tusks on an elephant), where internal "priors" override explicit negative constraints.

Quantitative Results

The failure rates across models were remarkably high:

  • Inaccuracy Rates: Overall inaccuracy was significant: GPT-4 (72%), Copilot (82.67%), and Gemini (60% in English).
  • The "Tusk" Failure: For the prompt "elephant with no tusks," models failed 100% of the time across almost all languages.
  • Gemini’s Language Gap: While performing best in English, Gemini failed to generate images for Hindi and French prompts entirely, citing language learning limitations.

Conclusion and Future Work

The study concludes that until this "blindness" is addressed, LLMs cannot be fully trusted for applications requiring high-precision image generation or strict adherence to safety-related exclusions. The authors propose implementing a "negation context-aware" reinforcement learning feedback loop to train models to verify visual outputs against negative constraints. Future work may involve testing broader open-source models to determine if this is a universal transformer-specific issue.

Summary updated

Abstract

Foundational Large Language Models (LLMs) have changed the way we perceive technology. They have been shown to excel in tasks ranging from poem writing and coding to essay generation and puzzle solving. With the incorporation of image generation capability, they have become more comprehensive and versatile AI tools. At the same time, researchers are striving to identify the limitations of these tools to improve them further. Currently identified flaws include hallucination, biases, and bypassing restricted commands to generate harmful content. In the present work, we have identified a fundamental limitation related to the image generation ability of LLMs, and termed it The NO Syndrome. This negation blindness refers to LLMs inability to correctly comprehend NO related natural language prompts to generate the desired images. Interestingly, all tested LLMs including GPT-4, Gemini, and Copilot were found to be suffering from this syndrome. To demonstrate the generalization of this limitation, we carried out simulation experiments and conducted entropy-based and benchmark statistical analysis tests on various LLMs in multiple languages, including English, Hindi, and French. We conclude that the NO syndrome is a significant flaw in current LLMs that needs to be addressed. A related finding of this study showed a consistent discrepancy between image and textual responses as a result of this NO syndrome. We posit that the introduction of a negation context-aware reinforcement learning based feedback loop between the LLMs textual response and generated image could help ensure the generated text is based on both the LLMs correct contextual understanding of the negation query and the generated visual output.
BibTeX
@misc{nadeem2024negation,
  author = {Mohammad Nadeem and Shahab Saquib Sohail and Erik Cambria and Björn W. Schuller and Amir Hussain},
  title = {{Negation Blindness in Large Language Models: Unveiling the NO Syndrome in Image Generation}},
  year = {2024},
  eprint = {2409.00105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.00105},
}
Figure