Who Catches What the Drafter Never Saw? Omission-Aware Annotation for AI-Image Hallucination Benchmarks

AI image generators now produce photographs whose textures and lighting look flawless but whose content is wrong: a hand with six fingers, a refrigerator standing in a forest, a cup fused into a table. Benchmarks for detecting these "semantic hallucinations" are expensive to label, so the largest ones are built by a model that drafts candidate errors and a human who accepts or rejects each draft. This note argues that the workflow has a blind spot built into its structure. An error that the drafting model never noticed is never shown to the human. It never enters the labels, and a model trained or scored on those labels is never penalized for missing it. The failure begins in the drafter, carries through the human check, and ends in the evaluation metric.

We propose to measure this effect directly and to repair it cheaply. The measurement compares draft-then-verify labels against independent references, broken down by how small the defect is and whether the drafter ever named the object involved. The repair adds a human review step that does not depend on the drafter: annotators look only at image regions that no draft covers. The method itself is modest. The case for doing the work is that it tests whether current benchmarks systematically overrate detectors on exactly the small, unparsed defects that people notice and that existing scores ignore.

Core Claim

Our hypothesis is that, in model-assisted annotation of AI-image hallucinations, the dominant unmeasured error is omission rather than wrong acceptance. We further hypothesize that omissions concentrate in small defects and in objects the drafter's first stage failed to list. Accept/reject verification cannot fix this, because people check what they are shown. A review of uncovered regions, independent of the drafter, should recover most omitted defects at a fraction of the cost of annotating from scratch. If the hypothesis holds, recomputing benchmark scores against repaired references should change how detectors rank on small defects, and quality predictors conditioned on the repaired defect sets should agree better with human opinion scores.

Three findings in the file motivate this. Semantic Visual Anomaly Detection and Reasoning in AI-Generated Images builds the AnomReason benchmark with a three-stage GPT-4o pipeline (AnomAgent). The pipeline first lists the entities in an image, then mines anomalies for those entities, and then has humans accept or reject the results. By construction, anything absent from the first-stage entity list cannot become a label. AGIDefect-4K reports that more than 85% of defects occupy under 5% of the image, and that zero-shot GPT-4o and Gemini reach detection AUCs of only 0.29–0.38 because they are biased toward judging polished images as fine. Taken together, these results suggest that a GPT-4o-style drafter will miss many small defects. The scene-graph literature, where a "scene graph" is a list of objects plus the relations between them, constrains what a fix can look like. CURVE and HostSG clean noisy graphs by pruning uncertain edges, and both state that no downstream processing can recover an object the upstream detector missed. CURVE adds that uncertainty-based pruning risks discarding rare genuine anomalies, and hallucinations are precisely such rare anomalies. Any repair therefore has to get new evidence back from the pixels. Reasoning over the existing draft is not enough.

Research Program

The first component is an omission audit that needs no new human labels. AGIDefect-4K was annotated by 20 experts working without model drafts, and it provides pixel masks. It can therefore serve as an independent reference that the drafter's blind spots have not shaped. We will run an entity-first drafter on these images. One version reproduces the AnomAgent prompting structure with GPT-4o, and an open version uses Qwen2.5-VL-7B. We then simulate an ideal verifier that accepts every draft overlapping a ground-truth mask. Under this generous verifier, any remaining miss is a pure omission. We report omission rate as a function of defect area, defect type (anatomical, physical, contextual), and whether the drafter's entity list named the affected object. The last split gives a direct test of the cascade: if errors on objects that were never listed account for most omissions, the upstream parsing step is the bottleneck.

The second component is a controlled human study on 300 images, drawn half from AGIDefect-4K and half from AnomReason, with three annotators per image under each condition. In the blank-slate condition, people mark every defect without any draft. In the draft-verify condition, they accept or reject AnomAgent-style drafts, which is the current practice. In the draft-verify-plus-uncovered-review condition, they verify the drafts and then examine a sequence of image tiles that contain no accepted draft, marking anything wrong. The annotation interface will log time per image in every condition. The reference set for each image is the adjudicated union of all defects found in any condition. This avoids treating any single condition as ground truth, which would bias the study toward that condition. The literature lists anchoring on model drafts as an open problem in the summary we work from, but we found no study that quantifies it for this task. The difference between the blank-slate and draft-verify conditions provides that measurement.

The third component is automated recovery, which tests whether the uncovered-region step can be done partly by machine. The first variant prompts the same multimodal model, meaning a language model that reads images, on multi-scale crops instead of the whole image. A second variant prioritizes crops using the defect heatmap from AGIDA, the defect-aware quality model introduced with AGIDefect-4K. A third, lighter variant borrows from ActLocalizer, which showed that corrections on 5% of video segments can be spread across hundreds of others and cut annotation time by 78%. In our version, when an annotator adds a missed defect of some type, such as hands, the drafter is re-prompted with that example on other images that contain the same entity type.

The fourth component traces consequences downstream. On a re-annotated subset of the AnomReason test split, we recompute SemAP for several detectors. SemAP is the paper's precision-style metric, which matches predicted explanations to human ones using BERTScore. We then check whether the ranking of detectors changes. On AGIDefect-4K, we test whether a quality predictor that sees repaired defect sets tracks human opinion scores better than one that sees draft-only sets. This connects the omission problem to the gap between automated scores and human judgment: AGIDefect-4K shows that people penalize anatomical defects far more than background ones, so missing a small hand defect should cost more agreement with humans than missing a background flaw.

Baselines

The naive reference point is draft-verify annotation exactly as practiced in AnomReason, with the same drafter, the same prompts, and the same accept/reject interface. Every recall, time, and ranking comparison is made against it. The method has failed if the uncovered-region review does not add recall beyond this condition.

The human upper reference is blank-slate expert annotation in the style of AGIDefect-4K. It is the obvious answer to omissions and also the most expensive. It runs on the same images with the same annotator pool, and time is logged. The proposal beats it if repaired annotation reaches comparable recall at substantially lower time per image. As a pre-registered target, repaired annotation should reach at least 90% of blank-slate recall at no more than half its time. No reported result was found on Lacuna for the annotation time per image in AGIDefect-4K, so the time baseline must come from our own logging.

The strongest automated detector baselines are the following. AnomReasonor-7B, a LoRA-tuned Qwen2.5-VL-7B from the AnomReason paper, reports SemAP_Full of 0.5162 on AnomReason and 82.61% accuracy on the deepfake split. AGIDA reports detection AUC 0.65 and quality-prediction SRCC 0.800 on AGIDefect-4K. Both can serve as stronger drafters as well as detectors to be ranked. Including them tests whether better drafting alone closes the omission gap. Each is run from released checkpoints if available, otherwise retrained under its authors' recipe on the same training split, with no extra tuning budget beyond what we give our own variants. For the quality experiment, the proposal beats AGIDA if conditioning on repaired defect sets raises SRCC above the 0.800 that AGIDA reports, on the same test split.

Two ablations isolate the key ingredient. Matched-time review shows annotators an equal number of random tiles that do overlap drafts, which separates "showing uncovered regions" from "spending more time." Free-text prompting replaces the tile review with a single "did we miss anything?" question at the end of verification, which is the cheapest possible omission check.

Why not just…?

Why not just annotate from scratch? If budget were unlimited, blank-slate annotation would be the right answer, and we use it as a reference. Model-assisted drafting exists because it is much faster: the Dresden OCT work reports a 15x speedup (Dresden in vivo OCT). Our question is whether that speedup can be kept without inheriting the drafter's blind spots. The human study measures both sides of the trade-off on identical images.

Why not just use a better drafter, such as AnomReasonor-7B or AGIDA? This is a real threat, and we treat it as a direct test. A fine-tuned drafter will omit less. However, AnomReasonor-7B was trained on labels produced by the entity-first pipeline, so its omissions may correlate with the very gaps we are measuring. AGIDA's detection AUC of 0.65 also leaves substantial room. If the fine-tuned drafters recover small, unlisted defects at close to blank-slate rates in the audit, then the human review step is unnecessary, and the contribution reduces to the audit together with the recommendation to use a tuned drafter.

Why not just tile the image and prompt the model on each crop? This is the most dangerous alternative. It directly targets the small-area problem, and it may recover much of what we attribute to human review. It is included as the first automated variant. If tiling alone matches the human uncovered-region review on recall, we will report that, and the proposal becomes an audit plus a tiling recommendation. Two caveats apply. Tiling multiplies drafting cost, and the AnomReason pipeline already consumed 4.17 billion tokens. Tiling also does not remove the positive bias that AGIDefect-4K documents, so cropping may make defects visible without making the model willing to flag them.

Why not just prune or reweight the scene graph automatically? CURVE and HostSG show that graph-level cleanup cannot recreate missing objects, and CURVE warns that uncertainty pruning removes rare, high-variance relations. Hallucinations belong to that category. Graph refinement is therefore the wrong tool for omissions, and we do not compete with it.

Proposed Experiments and Decisive Checks

The audit on AGIDefect-4K comes first, because it is cheap and can kill the idea. We report omission rate under the ideal verifier, split by defect area (below versus above 5% of the image) and by whether the affected object was in the drafter's entity list. If omission among small or unlisted defects is not clearly higher than among large or listed ones, the cascade hypothesis is false and we stop.

The human study then tests anchoring and repair. The decisive comparison is recall against the adjudicated union reference across the three conditions, with time per image as the cost axis and agreement between annotators reported for every condition. The study supports the proposal if draft-verify recall on small defects falls meaningfully below blank-slate recall, and if uncovered-region review recovers most of that gap while beating the matched-time and free-text ablations. Disagreement between annotators over borderline stylization is expected, since the AnomReason paper notes it. Adjudication guidelines will separate "intentional style" from "defect" before annotation begins.

The consequence experiments test whether any of this matters outside annotation. On the repaired AnomReason subset, we report SemAP for each detector under original and repaired references, and Kendall's tau between the two rankings. A tau well below 1, concentrated on small defects, would show that current scores reward the drafter's blind spots. On AGIDefect-4K, we compare SRCC with human opinion scores for quality predictors conditioned on draft-only versus repaired defect sets, against AGIDA's reported 0.800. A gain there would show that the omitted defects are the ones people actually care about, which ties the annotation pipeline's upstream blind spot to the gap between automated and human quality judgments.