Subtracting the Scene: Two-Stage Decoding Against Context-Driven Object Hallucination

Vision-language models, meaning chatbots that read an image and write text about it, often name objects that are not in the picture. They put a refrigerator in a photo of a kitchen counter, or a mouse next to a keyboard. Our premise is that this is a failure familiar from hierarchical classification. In a two-stage classifier, a first stage picks a coarse category and a second stage picks the fine label given that category. If the first stage dominates, a wrong or merely generic coarse guess dictates the fine answer. We argue that these models implicitly run such a pipeline. They form a coarse guess about the scene ("kitchen"), then name objects that the scene makes likely, and that conditioning overrides weak pixel evidence for the object itself.

The proposal is to make the two stages explicit at decoding time and use them for subtraction rather than routing. In stage one, the model itself writes a short scene-only description of the image. In stage two, whenever the model is about to emit an object word, we compare how likely that word is given the full image with how likely it is given only the scene description. We down-weight words whose probability the scene alone already explains. The method needs no training and no external object detector, and it adds only one cheap text-only forward pass at noun positions. Its central claim can be tested directly: object hallucinations are mostly "context-explained," and removing the coarse stage's prior reduces them more than generic contrastive decoding does.

Core Claim

Evaluating Object Hallucination in Large Vision-Language Models introduced POPE, which asks yes/no questions about whether an object is present. It showed that hallucinated objects track co-occurrence statistics: accuracy for MiniGPT-4 drops from 78.86 F1 on random negatives to 71.37 on adversarial negatives, which are chosen to co-occur with objects truly present. MERLIM found that models keep asserting objects after those objects are digitally erased, with an average precision drop of about 8%. Its authors read this as answering from global scene context rather than localized evidence.

Both findings motivate the proposal. Both are consistent with the hierarchical-classification account of a coarse stage whose prior leaks into fine decisions, the error-cascade problem documented in two-stage systems from acoustic scene classification to multimodal taxonomy classification.

We claim two things. The first is a measurable fact: a large share of hallucinated object tokens receive probability under a scene-only text input at least as high as under the actual image. The second is a method claim: contrasting against that specific scene-only input reduces adversarial-split and erased-object hallucination more than contrasting against a noised image or no image at all. It should do so at matched cost, without lowering recall of objects that are really present.

Concretely, let $x$ be the image, $s$ the model's own scene description, $y_{<t}$ the text generated so far, and $w$ a candidate object word. At noun positions we score

$$\text{score}(w) = (1+\alpha)\log p(w \mid x, y_{<t}) - \alpha \log p(w \mid s, y_{<t}),$$

where $\alpha \ge 0$ sets the strength of the subtraction. As in standard contrastive decoding, we apply this only to words that already have reasonable probability under the full image, so the model is never pushed toward words the image does not support.

The lesson we borrow from hierarchical classification is to route softly rather than through a single hard coarse decision. We therefore sample several scene descriptions and average the scene-only term over them. This way a wrong stage-one guess dilutes the subtraction instead of corrupting it.

Relation to Prior Work

The closest work also uses coarse-to-fine structure against hallucination, but with a different role for the coarse level. GTHM builds a three-level tree of image crops (global event, object relations, single entities) using the external detectors Grounding DINO and RAM. It weights contrastive decoding by a Shapley value, a measure of how much each view contributes to the current token. HALC contrasts detector-located crops at several zoom levels against the full image and runs image-text-matching beam search. Both treat the coarse view as additional evidence. We treat it as the source of the prior to be removed. Both also need a detector; we have none, and so we avoid the fallback to random sampling that HALC reports when its detector fails.

PerceptionCLIP is the closest in spirit. It is a two-step zero-shot CLIP classifier that first infers context attributes and then classifies the object conditioned on them. We reverse the use of context: PerceptionCLIP conditions on it to sharpen a discriminative label, while we subtract it to stop a generator from naming context-typical objects. We also derive the context from the model's own free-text description rather than a hand-defined attribute list.

TTC supplies evidence that explicit top-down structure fixes inconsistencies that model scale does not. On MEP-3M, an e-commerce product dataset with a three-level category tree, it raises mPLUG-Owl's finest-level accuracy from 54.04% to 72.03%. However, it is a fixed classification head over a hand-built taxonomy, not an intervention in open-ended text generation.

CHiP and MMA apply hierarchy at training time. CHiP uses preference optimization at the response, segment and token level; MMA uses multi-level alignment for video models. CHiP constrains our design by documenting that strict grounding makes models terse. For that reason we treat recall of true objects and caption length as primary metrics, not afterthoughts.

The representation-steering methods SHARP and AFTER edit middle-layer activations. They are strong training-free competitors, but they do not identify which prior is being removed.

Research Program

The work has three parts, and the first can kill the project cheaply.

The first part is a diagnostic. For LLaVA-1.5, LLaVA-Next and MiniGPT-4 we generate COCO captions and label each object word as hallucinated or grounded using CHAIR's ground truth (CHAIR measures the share of captions, and of mentioned objects, that are absent from the image). For every object word we compute a context ratio: its log-probability given the scene-only description minus its log-probability given the image. We do the same on MERLIM's inpainted images, comparing each erased object before and after erasure.

The hypothesis predicts three things. Hallucinated objects should have much higher context ratios than grounded ones. Their context ratios should be higher on POPE's adversarial split than on its random split. After an object is erased, its image-conditioned probability should fall toward its scene-only probability rather than toward zero. If hallucinated and grounded objects have similar context ratios, the hypothesis is wrong and we stop.

The second part builds the decoder. Stage one prompts the model for a one-sentence description restricted to place and setting. Stage two applies the scoring rule above at positions where the model's next token is likely to begin a noun. We detect these from the model's own probability mass on nouns, so no external part-of-speech tagger is needed at decoding time. The scene-only pass contains no image tokens, so it is far shorter than the full pass. We report its cost directly.

The third part tests error recovery. We deliberately corrupt stage one by substituting a wrong scene description, or one sampled from a mismatched image. We then measure how much a single hard description degrades performance compared with averaging over several sampled descriptions. This tests the hierarchical-classification claim that soft routing contains cascade errors.

Baselines

The naive reference point is plain greedy decoding on the same model and prompts. The file reports LLaVA-1.5 at 84.13% POPE accuracy (RITUAL paper) and 52.8 CHAIR_s (SHARP paper). It reports LLaVA-Next at 79.5% on the POPE adversarial split (SHARP paper).

The most direct competitor is image contrastive decoding with a destroyed-image negative: VCD, which uses a noised image, and its no-image variant. Our method differs from these only in what the negative input contains. For VCD the file gives only latency, 13.97 seconds per sample on LLaVA-Next as reported in SHARP; there is no reported accuracy result on Lacuna for VCD. We will rerun VCD with the same $\alpha$ search, the same plausibility cutoff and the same noun-position gating as our method. This makes the comparison isolate the negative input.

The strongest training-free numbers in the file are the following. RITUAL, which contrasts the original image with a randomly transformed copy, raises LLaVA-1.5 to 88.87% POPE accuracy. SHARP, which steers activations, reaches 82.8% on the POPE adversarial split with LLaVA-Next and lowers LLaVA-1.5 CHAIR_s to 34.8. HALC lowers MiniGPT-4 CHAIR_s from 30.87 to 17.80. GTHM reports a 13.1% relative CHAIR_s reduction over HALC.

We will reproduce RITUAL, SHARP and HALC with their released settings on identical images, and give each the same validation budget for tuning. Beating them means two things. First, on the POPE adversarial split and on MERLIM erased objects, we must exceed the best of these at equal or lower wall-clock time per sample. Second, on CHAIR we must achieve lower CHAIR_s without reducing ground-truth object recall or mean caption length by more than a few percent.

The ablation that isolates our key ingredient replaces the negative input while holding everything else fixed. We compare four negatives: the text-only scene description, a heavily blurred "gist" image that keeps scene layout but destroys local detail, a noised image, and an empty prompt with no image. If the scene-based negatives (text or blur) do not beat the noised and empty negatives on the adversarial split, the coarse-stage account adds nothing over generic contrastive decoding.

Why not just…?

Why not just use VCD or no-image contrastive decoding? This is the real threat. A model with no image still carries language co-occurrence priors, and it may remove most of what a scene description removes. Our prediction is narrower: the empty negative removes a generic prior (frequent words such as "person"), while the scene negative removes the conditional prior that drives adversarial-split errors. The four-way negative ablation settles this. If the empty negative matches the scene negative on adversarial POPE and MERLIM, we will report that the diagnostic finding stands but the decoding contribution does not.

Why not just prompt the model to describe the scene first and then list objects? Chain-of-thought prompting makes the coarse stage explicit, but it feeds the scene guess forward as conditioning. On our account, that strengthens the very prior that causes hallucination. MERLIM also reports accuracy swings of up to 26.5% across rephrasings of the same prompt, so prompting alone is fragile. We will run this scene-first prompting baseline under five paraphrases and report the spread. Our prediction is that it raises adversarial-split hallucination, which would itself be evidence for the cascade account.

Why not just steer activations as SHARP or AFTER do? Steering is fast and strong. Its vectors, however, are fitted once from offline data, whereas the scene prior differs from image to image. We compare directly and also try combining the two. If SHARP alone matches us on the adversarial split, the per-image scene subtraction is redundant. We will say so, keeping only the diagnostic as the contribution.

Why not just use a detector-based method like HALC or GTHM? Both perform well, but they inherit their detector's failures, and HALC runs roughly 2.4 times slower than greedy decoding without parallelization. We report the hallucination reduction obtained per unit of latency. We also report results on the subset of images where Grounding DINO misses the queried object, which is exactly where we expect detector-free subtraction to help.

Decisive Checks

We will run all methods on POPE (random, popular and adversarial splits over COCO, A-OKVQA and GQA), CHAIR on COCO captions, MERLIM's inpainted and false-relation sets, AMBER for long-form generation, and MME for the existence, count, position and color subtasks, which check for collateral damage.

Three results decide the project. The first is the diagnostic: the context-ratio gap between hallucinated and grounded objects. The second is the negative-input ablation on the adversarial split and on erased objects. The method is supported only if both scene-derived negatives beat the noised and empty negatives, and the gain is concentrated on adversarial and erased cases rather than random ones. The third is the honesty-versus-verbosity check: CHAIR_s must fall while object recall and caption length hold.

A secondary check reuses the observation from Why LVLMs Are More Prone to Hallucinations in Longer Responses that hallucination grows with response length. We will plot the context ratio of hallucinated words against their position in the response. If late hallucinations are increasingly context-explained, a subtraction targeted at the scene prior should help most in long captions, and AMBER will show whether it does.