Tools that find where a model fails automatically, such as error-separating directions in embedding space or language-model loops that write a sentence describing each failing group, are increasingly the way practitioners debug classifiers. Many of the most important failures of fake-image detectors are defined by something missing. A detector may wave through fakes that contain no faces, no fine texture, or no rendered text. Such failures are described in sentences containing "no," "without," or "lacks." The tools that check whether a description fits a group of images usually score it with CLIP, the contrastive image-text model. CLIP is known to read "a car without tires" as "a car with tires" (Seeing What's Not There). So the audit can misdescribe or reject exactly the failures that matter most.
We propose to measure this blind spot and to remove it. We build a testbed of freshly generated images with deliberately planted detector failures. Some failures are defined by presence of an attribute and some by its absence, in otherwise matched constructions. Because the images are new, no model can have memorized them, and the true failing group is known by construction. On this testbed we ask whether existing failure-discovery methods recover absence-defined failures as well as presence-defined ones. We then test whether a simple change closes the gap. The change is to rewrite every candidate description into affirmative attribute checks and apply the negation in code rather than inside the embedding.
Core Claim
Automated failure discovery inherits the negation blindness of the vision-language model used to describe and verify its findings. For forgery detectors this produces a measurable asymmetry. Planted failures defined by the absence of an attribute are recovered and correctly described markedly less often than failures defined by its presence, even when the underlying error geometry is equally separable.
The claim is new in two ways:
- Prior negation work is about retrieval, not auditing. It documents and patches negation failures in retrieval and multiple-choice settings. It has never examined the verification step inside an auditing pipeline.
- Prior discovery work assumes an affirmation-neutral checker. Discovery methods implicitly treat their description checker as unbiased. None has been evaluated on forgery detectors with ground-truth slices.
A second, smaller claim concerns contamination. Fresh generations give an honest estimate of the cross-generator gap that static public sets cannot. Our planted design lets us check whether the failures found on public GenImage images survive on new images from the same generator.
Research Program
Planting failures. The first component is a planted-failure testbed. We follow the logic of Towards a More Rigorous Science of Blindspot Discovery, which argues that discovery methods can only be validated against known blind spots. We train detectors on curated mixtures in which a spurious attribute is tied to the label.
- Presence slices. Attribute A, for example visible text or a human figure, appears only in real training images. The detector learns to call fakes containing A "real."
- Absence slices. Attribute B appears in every training fake. The detector learns to call fakes lacking B "real."
The two constructions are symmetric: the same attributes are swapped between presence and absence roles across runs. Any asymmetry in discovery can therefore be attributed to the description and verification stage rather than to the attribute itself.
Generating and labeling test images. Training data come from the GenImage Stable Diffusion V1.4 split and its real ImageNet counterpart. Test fakes are generated fresh from open generators, using prompts that set attribute presence. Membership is confirmed by an open-vocabulary detector plus a human audit of a sample.
Controlling salience. How and where does CLIP process negation? showed that VALSE negation accuracy tracks object size (r = 0.32). We therefore match the attribute's area across presence and absence conditions so that visual salience cannot masquerade as a negation effect.
Running existing discovery methods unchanged. The second component runs three families of failure discovery on each planted detector:
- Error-separating directions. We use the linear-SVM method of Distilling Model Failures as Directions in Latent Space. It finds a direction in CLIP image space separating errors from correct predictions, then names it with the best-matching captions.
- Cluster-then-explain loops. We adapt DISCERN to images. A captioning model and a language model propose predicates for each error cluster, and an evaluator checks them against in-cluster and out-of-cluster images.
- Internal-representation clustering. Following Targeted Data Generation, we cluster the detector's own features, then describe the clusters with the same explainer.
We record two things separately: whether the right images were grouped, and whether the right sentence was produced and accepted. This separation is essential. It tells us whether negation blindness corrupts the geometry or only the language layer.
The proposed fix. The third component is a verification step we call symbolic affirmative decomposition. A language model rewrites each candidate predicate into atomic affirmative attribute queries joined by explicit logical operators. For example, "fakes with no visible text" becomes NOT(contains text). Each atom is scored affirmatively per image, and the logic is applied in code. The same decomposition is used to rank candidate captions for SVM directions. The fix needs no retraining. That matters because Seeing What's Not There reports that fine-tuning on negation hard negatives causes forgetting.
Contamination audit. The fourth component reruns the cross-generator evaluation of a strong language-guided detector on fresh images from the same generators and prompt distributions as public GenImage. A drop on fresh images would indicate the static score is inflated. Matched slices across the two image sources would indicate the discovered failures are real rather than artifacts of memorized images. This mirrors the logic of DyVal and MMBench-Live, applied to image forensics.
Baselines
Naive reference. Ranking test fakes by detector confidence and taking the lowest-confidence group as "the failure." This is what a practitioner does with no tooling. We run it on the same planted detectors and score it with the same slice-precision metric.
Strongest existing discovery method. The SVM failure-direction method of Distilling Model Failures as Directions in Latent Space. It is the strongest competitor because it needs no clustering thresholds and works directly in CLIP space, where our attributes live. Its reported result is 89% of flagged images coming from the hard subpopulation on biased CelebA, where 33% of the data was hard. We run it with its published recipe on the same validation embeddings and the same number of flagged images as every other method. Beating it means equal or better slice precision on presence slices, and a statistically significant gain on absence slices. There is no reported result on Lacuna for this method on forgery detectors; we establish one.
Language-loop baseline. DISCERN with its original evaluator, given the same language model, the same number of refinement rounds, and the same clustering distance threshold sweep. DISCERN reports roughly 3% absolute accuracy gain on AGNews after augmentation, and at least a 10% error drop in problem clusters. Nothing is reported for images or forgery.
Strongest negation fix. The anchor-adjusted projection subtraction of Seeing What's Not There, applied to the evaluator's text embeddings. It reports raising zero-shot CLIP on NegBench-COCO multiple choice from 24.7% to 72.5%. It is the strongest training-free negation repair available and the most direct competitor to our decomposition.
Detector being audited. A CLIP-based detector in the style of C2P-CLIP, which reports 95.8% accuracy on GenImage and 93.8% mean accuracy on UniversalFakeDetect. Its GenImage figure is the number our contamination audit checks against fresh images.
Ablations. Three ablations isolate the key ingredient:
- Decomposition with symbolic logic removed, so atoms are re-joined into a single negated sentence and scored by CLIP.
- Decomposition with CLIP scoring replaced by a multimodal language model answering yes/no on the original negated sentence.
- The presence and absence attribute roles swapped, to confirm that the asymmetry tracks the role and not the attribute.
Why not just…?
Why not just fine-tune the auditor's CLIP on negation hard negatives? Seeing What's Not There reports that NegCLIP-style fine-tuning causes forgetting and increased category confusion. An auditor whose general attribute recognition degrades would mis-score presence slices too. We include a NegCLIP-tuned evaluator as an extra arm. If it closes the absence gap without hurting presence-slice precision, that is a real result and we would report it as competitive.
Why not just ask a multimodal language model whether each predicate holds? This is the most serious threat. A capable model answering "does this image contain no text?" might handle negation well enough that decomposition adds nothing. Our second ablation tests exactly this. If yes/no prompting on raw negated predicates matches symbolic decomposition, the contribution narrows to the diagnosis and the testbed, and we will say so. We expect some residual gap, because On Classifying whether Two Texts are on the Same Side of an Argument and ALIGN-SIM document that language models still conflate statements with their negations.
Why not just sort fakes by detector confidence? Distilling Model Failures as Directions in Latent Space found raw confidence to be a poor indicator of systematic bias. Hidden stratification shows models can be equally confident on failing and prototypical groups. Planted spurious shortcuts typically yield confident errors, so we expect this baseline to fail on both slice types. The naive-reference arm measures this directly.
Why not just evaluate on GenImage and Celeb-DF as everyone does? Static sets cannot tell us which images belong to a failing group. They may also have been seen during pretraining of the CLIP backbones used by both detector and auditor. Fresh generation solves both problems at modest cost, and the contamination audit quantifies how much the choice matters.
Proposed Experiments and Decisive Checks
Scale and metrics. We train detectors under 12 planted configurations: six attributes, each used once as a presence slice and once as an absence slice, with three seeds each. Every discovery method is applied under a matched budget. We report three metrics:
- Slice precision: the fraction of flagged images inside the planted slice.
- Slice recall at fixed budget.
- Description accuracy: whether the accepted predicate is logically equivalent to the planted rule, judged by two annotators blind to method.
First decisive check: does the asymmetry exist? We compare absence-minus-presence differences in description accuracy for the unmodified DISCERN-style and SVM-captioning pipelines. A gap of under a few points, with overlapping seed intervals, would falsify the core claim. If the gap is large in description accuracy but not in slice precision, negation blindness lives in the language layer, not the geometry. That finding would still change how these tools should be built.
Second decisive check: does decomposition close the gap? We test whether symbolic decomposition closes the gap more than projection subtraction does, without lowering presence-slice accuracy. We also test whether it beats yes/no prompting of a multimodal model.
Third check: contamination audit. We report the audited detector's accuracy on public GenImage versus fresh same-generator images, and the overlap between slices discovered on each. A large accuracy drop with little slice overlap would mean static forgery benchmarks are misleading both the headline number and the debugging that follows from it.
References
- Seeing What’s Not There: Negation Understanding Needs More Than Training
- Towards a More Rigorous Science of Blindspot Discovery in Image Classification Models
- GenImage: A Million-Scale Benchmark for Detecting AI-Generated Image
- How and where does CLIP process negation?
- Distilling Model Failures as Directions in Latent Space
- DISCERN: Decoding Systematic Errors in Natural Language for Text Classifiers
- Targeted Data Generation: Finding and Fixing Model Weaknesses
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models
- C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection
- On Classifying whether Two Texts are on the Same Side of an Argument
- ALIGN-SIM: A Task-Free Test Bed for Evaluating and Interpreting Sentence Embeddings through Semantic Similarity Alignment
- Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging