Isolate, Then Ask: Re-Querying Cropped Objects to Correct Attribute Hallucinations

Vision-language models (systems that read an image and write text about it) often name the right objects but give them the wrong properties. They describe a blue bike as red, a sitting dog as running, or swap the colors of two neighboring objects. Most hallucination fixes target the simpler question of whether an object is there at all. Several papers say outright that attribute errors are left for future work. We propose a simple, testable mechanism for attribute errors. The model first writes a draft caption. A deterministic parser pulls out each object–attribute claim. For each risky claim, the system shows the model the object alone, cropped or with the other objects masked out, and asks about the attribute again. If the answer about the object alone disagrees with the draft, the claim is flagged and revised. The bet is that many attribute errors are binding errors. The model sees "red" and "bike" somewhere in the scene but attaches the color to the wrong object, and removing the other objects from view removes the chance to mis-bind.

What makes this a research question rather than an engineering trick is a specific hypothesis. Asking the same model the same thing again on the same full image is a poor check for attribute errors, because the model tends to repeat the same wrong binding. Changing what the model sees should break that self-agreement. We test this with a pipeline built from swappable parts: claim extraction, a risk trigger, an object locator, an isolation operation, a verifier, and a reviser. Swapping one part at a time lets us show which ingredient produces the gain, and lets us reject the idea cleanly if it does not.

Core Claim

The closest existing work is Logical Closed Loop (LogicCheckGPT). It verifies objects by asking the model for an object's attributes and then asking which objects have those attributes. It relies on a "consistency gap": the model is self-consistent about real objects and contradicts itself about invented ones. That paper uses attributes only as a stepping stone to check whether an object exists, and it names attribute hallucination as a limitation. The obvious extension, running the same loop in reverse to check attributes, inherits an assumption we think fails for attributes.

When an object really exists, the model's attribute errors are often systematic rather than random. Two lines of evidence point this way. Common Data Properties Limit Object-Attribute Binding in CLIP traces binding failures to training-data properties, especially captions that mention only the salient object, rather than to model scale. Correcting Visual Blur Induced by Attention Distraction shows that hallucinated tokens come with attention spread over irrelevant regions. A model that binds attributes wrongly for a stable data-driven reason will answer a rephrased question the same wrong way.

Our claim is that the useful refinement for attributes changes the visual input, not just the question. For a claim pairing object $o$ with attribute $a$, we define an isolation gap

$$\Delta(o,a) = \log p_\theta(a \mid I, q_{o}) - \log p_\theta(a \mid I_o, q_{o}),$$

where $p_\theta$ is the model's probability of answering with attribute $a$, $I$ is the full image, $I_o$ is the image with everything but object $o$ cropped or masked away, and $q_o$ is a short question about $o$'s attribute ("What color is the bike?"). A large positive gap means the attribute is supported only when other objects are in view. That is the signature of a binding error. We predict this gap separates true from hallucinated attribute claims much better than plain re-asking on the full image. We also predict the advantage is largest for intrinsic attributes such as color, material, and shape, and smallest for attributes that need context, such as actions and counts.

Research Program

The system has a draft-then-revise structure. This follows the "End-Revise" result of Retrieving, Rethinking and Revising, where reformulating a search query after seeing a draft beat rewriting it beforehand, because the draft exposes exactly what needs checking. Here the draft caption plays that role: it tells us which bindings to verify.

Claim extraction is deterministic. It uses adjective–noun and verb–subject parsing in the style of AMBER, which isolates attribute hallucinations without an LLM judge. This avoids LogicCheckGPT's reliance on an external language model, along with that model's cost and its own hallucinations.

To control cost, a trigger module decides which claims to check. Our default trigger is a linear probe on middle-to-late hidden states. Activation Steering Decoding reports that hallucinated content is linearly separable there with about 82.5% accuracy. We prefer a hidden-state probe to attention-entropy triggers because the latter need full attention matrices. That requirement is the stated reason Modular Attribution and Intervention cannot run with fast attention kernels such as FlashAttention.

The locator finds the object's region in one of two ways. The first is the model's own logit-lens grounding, reused from GLSim, which projects object tokens onto the top-$K$ image patches. The second is native box output on backbones that support it. The isolation module offers two operations, and separating them is central to the analysis. Crop-and-zoom enlarges the object. Mask-at-original-resolution greys out other objects but keeps the pixel scale unchanged. Only masking cleanly tests the "remove the competing objects" hypothesis, because cropping also adds effective resolution, which is the effect behind GUI-Spotlight's iterative zooming.

The verifier is swappable. It can be the same vision-language model, a CLIP-style zero-shot attribute classifier applied to the isolated region, or a small verifier model of the kind OmniEvaluator shows can stand in for large judges. The reviser either substitutes the verified attribute word directly or prompts the model to rewrite the sentence with the correction.

The interfaces follow the abstract-class style that Universal Causal Evaluation Engine uses to separate data generation from estimation. Our purpose is the same: running a factorial ablation without glue code between conditions. Seeds and code versions are pinned in the manner of DeepDIVA. Modularity is a methodological tool here, not the contribution.

Baselines

The naive reference is greedy decoding on the unmodified backbones, LLaVA-1.5-7B and Qwen-VL, which most cited mitigation papers use. For scale, AFIP's paper reports greedy sentence-level CHAIR of 55.4 on LLaVA-1.5-7B, which AFIP reduces to 16.8.

The ablation that isolates our key ingredient is attribute re-asking without isolation. It uses the same pipeline, the same trigger, and the same number of model calls, but asks $q_o$ on the full image $I$. We also include LogicCheckGPT adapted to attributes. That variant reverse-queries "which object is red?" and checks whether the answer returns to the original object, still on the full image. It is the strongest competitor because it is the published closed-loop method closest to ours, and it isolates the consistency-gap assumption we argue fails for attributes. Its reported strength is on existence: it raises POPE Adversarial accuracy on mPLUG-Owl from 50.67% to 82.00% (Logical Closed Loop). We found no reported result on Lacuna for any attribute-targeted version on AMBER or the multi-image object hallucination benchmark (MIOH).

The single-pass decoding baselines are Fox (Dismantling Pathological Shortcuts), which reports 81.93% POPE Adversarial on LLaVA-1.5, ASD, and AD-HH. They represent the cheapest strong alternative, even though their authors scope them to object existence. We found no reported result on Lacuna for any of them on AMBER's attribute dimension.

The training-based reference is Antidote, which applies preference optimization (DPO) on synthetic contrastive data. It reports CP-Bench F1 rising from 5.7% to 78.4% on LLaVA-1.5-7B.

All methods run on the same images and backbones. Hyperparameters, including our gap threshold and the baselines' head counts and gates, are tuned with the same search budget on a held-out split of AMBER images disjoint from the test images. Every comparison reports model calls and wall-clock time per caption.

We count our method as winning under two conditions. First, it must lower AMBER's attribute hallucination rate below the no-isolation re-query at matched calls, by more than the bootstrap 95% interval across three seeds. Second, it must not regress CHAIR or POPE on the same backbone.

Why not just…?

Why not just use a single-pass decoding fix like Fox, AFIP, or ASD? These methods are cheap and cut object hallucination substantially. However, their authors state that attribute errors, counting, and relations remain open, so they have not been shown to address our target. A plausible mechanism exists by which they might help anyway: sharpening attention could reduce mis-binding. We therefore run all three on AMBER's attribute split rather than assume the answer. If one of them matches isolation re-querying at a fraction of the cost, that is a real negative result for us, and we would report it as such.

Why not just ask the model again, LogicCheckGPT-style? This is the threat our design is built around. We measure it directly. Among AMBER attribute hallucinations, we compute the fraction for which full-image re-asking agrees with the hallucinated attribute. Our hypothesis predicts this fraction is high, while the same statistic for nonexistent objects is low, which is LogicCheckGPT's consistency gap. If full-image re-asking proves as discriminative as isolation, our central claim is false.

Why not just fine-tune with preference data, as Antidote does? Antidote shows large gains on false-premise questions. However, the CLIP binding study finds that hard-negative fine-tuning helps only marginally when training data has natural saliency bias. Fine-tuning also changes the model, whereas our method is training-free and backbone-agnostic. We report Antidote alongside our method on AMBER attributes. We also test whether the two combine, since an isolation check could run on top of a preference-tuned model.

Why not just crop each object and classify its attribute with CLIP, skipping the language model entirely? This alternative may work as well as our language-model verifier. If so, it supports rather than undermines our claim, since the claim concerns isolation, not who verifies. CLIP's documented binding weakness arises with multiple objects in view, and a single-object region removes that condition. We include CLIP-on-region as a verifier module and treat the comparison as a finding. If CLIP-on-region matches the language-model verifier, the recommended system becomes the cheaper one.

Proposed Experiments

The primary evaluation is AMBER's attribute dimension, which covers state, number, and action, in both its generative and discriminative forms. We report per-attribute-type results. The prediction is a large gain for state attributes such as color and material, and little or none for action and count, where cropping removes needed context. The second evaluation is MIOH's attribute-binding task, using only images that show a single scene, from the PACO and Synthetic Visual Genome sources. MIOH reports 49.1% overall accuracy for Qwen2-VL-7B, and we found no attribute-task number on Lacuna for the backbones we use. We additionally construct a controlled swap probe from MIOH and AMBER images in which two objects carry different values of the same attribute. Here a binding error is unambiguous. No-regression checks use CHAIR, POPE, and MME, so we can confirm that revisions do not introduce object errors or damage general ability.

Three checks decide the outcome. The first is the matched-call comparison between isolation and no-isolation re-querying on AMBER attributes: if isolation does not win, the idea fails. The second is the mask-versus-crop comparison, which tells us whether the gain comes from removing competing objects (masking wins or ties) or merely from extra resolution (only cropping helps). The third is the cost curve. We plot the attribute-error reduction against average extra model calls per caption as the trigger threshold varies, alongside an always-check setting and a random-trigger control. This shows whether the hidden-state probe makes selective checking pay off, in the way GUI-Spotlight found that most cases needed only one refinement step.

Finally, we audit localization failures separately. Errors where the locator picks the wrong region would mimic genuine binding errors, so we hand-label 200 flagged claims to estimate how often the isolation step is itself the source of error.