TL;DR
The new T2I-NSI benchmark reveals a persistent "negation blindness" in modern text-to-image (T2I) models, which frequently ignore instructions like "without" or "no." By evaluating nearly a million images across six major models, researchers found that while models almost never "invert" a prompt (doing the opposite), they overwhelmingly fail to suppress negated concepts, with even frontier closed-source models scoring remarkably low on negation sensitivity.
What Problem Does This Paper Address?
While modern T2I diffusion models can generate stunningly complex scenes, they often trip over simple logic. If you ask for "a desk without a lamp," there’s a high probability you’ll get a desk with a lamp. This "negation blindness" is a known issue in discriminative vision-language models (like CLIP), but its impact on generative models is more severe because users have no text output to inspect—only a final image that violates their constraints.
This is particularly risky for safety and compliance. Imagine an automated system generating a "weapon-free" scene that silently inserts a firearm because it ignored the word "no." Prior evaluations of negation were small-scale and hand-crafted; this paper introduces a systematic, large-scale framework to measure exactly how and why these models fail.
Methods
The authors introduce T2I-NSI (Negation Suppression Index), a benchmark that treats negation as a contrastive switch.
- Paired Prompts: Every negated prompt (e.g., "an unlit lamp") is paired with a canonical positive counterpart ("a lit lamp").
- Factorization: They categorize negation by linguistic form (lexical/semantic like "dirty/clean", morphological like "un-", and syntactic like "no/not") and target complexity (ranging from simple objects to complex relations and nested cases).
- Robust Metric: Instead of relying on a single model to judge the output, they use an ensemble of three "probes": CLIP (global semantics), InstructBLIP (fine-grained VQA), and GroundingDINO (localized detection).
- Probe-Side Isolation: To ensure the evaluation itself isn't "blind," the probes are only ever asked to look for the positive concept (e.g., "Is there a lamp?"). This avoids the known issues where the judge model might also misunderstand negation.
Datasets
The researchers built a massive corpus:
- 759,869 paired prompt atoms derived from a catalog of 5,082 unique objects.
- 916,272 generated images across six models:
- Open-weights: FLUX.2, Qwen-Image-2512, and Stable Diffusion 3.5 Large Turbo.
- Closed-source: GPT-Image-1.5, Gemini-2.5-Flash-Image, and Midjourney v7.
Key Insights and Ideas
- Under-suppression, not Inversion: The most striking "aha moment" is that models almost never do the opposite of what is asked (Inverse rate = 0.00%). Instead, they simply fail to remove the object. They treat "without a lamp" as if the "without" wasn't there.
- The Linguistic Hierarchy: Negation sensitivity isn't uniform. Lexical negation (using antonyms like "hidden" instead of "visible") is the easiest for models to handle. Syntactic negation (using "no" or "not") is the hardest.
- The Relational Trough: Models struggle most with negating relations (e.g., "not on the table"). This mirrors a "U-shaped" difficulty curve seen in other vision-language tasks, where very simple and very complex prompts are easier than mid-level relational ones.
- The Open-vs-Closed Gap: While closed-source frontier models (GPT, Gemini) are about 3.5x better at negation than open-weights models, their absolute performance is still very low, suggesting this is a fundamental architectural or data-level challenge for diffusion models.
Key Results
- Low Sensitivity Scores: On a scale where 1.0 is perfect negation awareness, open-weights models scored $\le$ 0.038. GPT and Gemini reached only ~0.09.
- Persistence of Concepts: In the open-weights tier, the negated concept was "Always Present" in 13-17% of cases and only partially suppressed in another 9-12%.
- Success is Rare: "Clean" successes (where the object appears in the positive prompt but is totally absent in the negative one) occurred in less than 0.3% of evaluations for open models.
- Human Corroboration: A blinded study with 9 annotators confirmed the near-absence of inversion and the high rate of concept persistence.
Limitations and Future Work
- English-Only: The current benchmark is limited to English prompts.
- Visually Grounded Negation: The study focuses on objects and relations that can be seen in a single image, excluding non-visual predicates.
- Probe Bias: While the ensemble metric is robust, it still inherits some biases from the underlying vision-language models used as judges.
- Future Directions: The authors suggest that future T2I training should move beyond single-prompt sets and incorporate paired-prompt training or specific "scoped-negation" paraphrasing to teach models the logic of "no."
Summary updated