Two Kinds of Hallucination: Training Vision-Language Models on Absent Look-Alikes and Absent Associates

When a vision-language model, meaning a model that reads an image and answers questions about it in text, claims that something absent is in the picture, current work treats every such error as the same failure. We think there are at least two different failures. In the first, the model sees a laptop and asserts a mouse, because mice and laptops appear together in text. The mouse is related to what is present. In the second, the model sees a cup and asserts a mug, because the pixels are ambiguous between two things of the same kind. The mug is similar to what is present. Lexical semantics has long kept these two relations apart: relatedness is co-occurrence in situations, similarity is shared kind or function. Multimodal hallucination research has not. The best-known hard-negative test builds its "adversarial" absent objects purely from co-occurrence statistics, and recent preference-training methods corrupt images without asking which kind of wrong answer they provoke.

Our proposal is to split negative training samples, meaning examples where the queried object is absent and the correct answer is "no" or "not there", into absent associates and absent look-alikes. We then test a sharp prediction. Associate errors come from the model leaning on its language prior instead of the image, which is the problem usually called modality dominance. Look-alike errors come from weak perception. If that is right, the two types of negatives should repair different things, a fixed negative budget should be allocated between them on purpose, and both training and evaluation should report them separately. The prediction also applies to the large-model judges now used to score open-ended answers, since those judges show the same text dominance.

Core Claim

Hallucinated objects can be divided by their semantic relation to the objects actually present, and the division tracks mechanism. A simple test tells the two apart. Replace the image with an exact text list of the objects it contains, then ask the same presence question again. If the model still affirms an absent associate (mouse, given "laptop, desk, keyboard"), its belief came from language co-occurrence. A look-alike error (mug, given "cup") should mostly disappear under the exact list, because the confusion lived in the pixels. We call this the text-substitution test.

Given a fixed budget of negatives per training example, we claim that associate negatives mainly reduce text-dominance errors and look-alike negatives mainly reduce perceptual errors. Neither transfers fully to the other. An allocation chosen from the text-substitution diagnosis of the base model should beat co-occurrence-only negatives at the same budget. If associate negatives alone match the mixed allocation on every split, the claim fails, and we will report that.

Research Program

Building relation-typed absence probes. We start from the construction used in Evaluating Object Hallucination in Large Vision-Language Models (POPE). POPE asks binary "Is there a [object]?" questions and draws absent objects at random, by frequency, or by co-occurrence with present objects. It identified both the yes-bias and co-occurrence priors as causes of hallucination. It did not separate co-occurrence from taxonomic similarity, so its adversarial split mixes the two. We keep its polling format because POPE showed it is stable across prompt wordings (standard deviation 0.78 versus 3.22 for caption metrics).

For each image in COCO and Visual Genome, we score candidate absent objects on two axes. The first is relatedness: co-occurrence with the present objects, using Visual Genome's 2.8M relation triples. The second is similarity: closeness in the WordNet hierarchy, in the spirit of the WordNet-derived non-distributional vectors of Faruqui et al. (2015), and checked on overlapping pairs against the similarity ratings of SimLex-999. We then keep four cells: high-related/low-similar (associates), high-similar/low-related (look-alikes), high on both (fork near knife), and low on both (random).

We confirm absence with an open-vocabulary detector plus human spot checks, following the LLM-plus-detector verification pipeline of Vision-Language Models Do Not Understand Negation. Look-alike candidates fail verification more often, since a mug may really be there, so their spot-check rate is higher. We extend the probes past yes/no to counting (zero-count questions) and to free-form referring phrases in the style of OmniLabel. OmniLabel already showed that plausible negative descriptions break detectors but never typed them by relation.

The text-substitution diagnosis. For each base model we measure affirmation rates on each cell under three inputs: the real image, a blank image, and the exact ground-truth object list given as text. This extends the Image Information Gain idea of Modality-Balancing Preference Optimization, which compares confidence with the real image against an all-zero image. A blank image removes both the evidence and the context that triggers the prior. The text list removes only the pixels and keeps the context, and that is what lets us separate prior-driven errors from perception-driven ones. As a secondary mechanistic check, we will look at whether associate errors coincide with low mid-layer attention entropy and heavy attention on text tokens. C³PO reports that profile (90–95% of attention on text) for dominance-driven failures in omnimodal models.

Training with a typed negative budget. We train with multi-negative preference optimization. This means teaching the model to rank a grounded answer above several wrong ones, using the Plackett–Luce objective of MISP-DPO. Here, though, the negatives are textual: answers or captions that assert an absent associate or an absent look-alike. MISP-DPO's negatives are perturbed images chosen for diversity in sparse-autoencoder latent factors, not typed by relation. MISP-DPO also found that performance degrades beyond three to five negatives per positive. That result sets our budget at four negatives and makes how to divide them the real design question. We compare all-associate, all-look-alike, uniform-mixed, and diagnosis-allocated mixtures. In the diagnosis-allocated mixture, the share of each type is set by the base model's text-substitution error profile on a held-out split.

Keeping the judge honest. Open-ended evaluation will use a model as judge, so we first audit the judge on the same typed split. MLLM-as-a-Judge found that multimodal judges score more accurately from text descriptions than from pixels, which is text dominance inside the judge. It also found that step-by-step reasoning can make judges invent visual details. The Silent Judge showed that text judges change verdicts on irrelevant cues while never admitting it (0% cue acknowledgment). Neither paper tested semantic-relation errors.

We inject single associate or look-alike hallucinations into otherwise correct captions and measure three things: how often the judge catches them, with the real image versus the blank image versus the object list; whether it gives partial credit to look-alikes; and whether its stated reasons mention the error. This audit is a constraint on our evaluation, not a second contribution. Headline numbers rest on deterministic presence checks. Judge scores are reported only on splits where the audit shows acceptable catch rates.

Baselines

The naive reference is the untrained base model, Qwen2.5-VL-7B and LLaVA-v1.5-7B, evaluated on every typed split. It tells us whether the split structure exists at all.

The first trained baseline is standard pairwise preference optimization (DPO) on RLHF-V-Dataset, which has over 5,000 human-feedback samples, with one negative per positive. This is the default practice that typed negatives must beat.

The second is co-occurrence-only negatives: four POPE-adversarial-style associates per positive. This is the strongest simple competitor, because it already targets the prior the field knows about. Beating it is the core test of the proposal.

The strongest existing methods are three. MBPO uses adversarial images from projected gradient descent plus online reinforcement learning on verifiable questions. It reports ObjectHal CHAIR_S falling from 14.1 to 7.4 on Qwen2.5-VL-7B and an MME gain of over 26 points. MFPO (Modality-Fair Preference Optimization) erases target objects with segmentation masks plus diffusion noise and reports a 40% improvement on Object HalBench for LLaVA-v1.5-13B. MISP-DPO reports a 30.09% average improvement over LLaVA-v1.5-7B across its suite. No per-benchmark POPE number is reported on Lacuna for MBPO, MFPO, or MISP-DPO, so we will rerun all three ourselves.

All methods get the same base checkpoints, the same RLHF-V source images, the same negative count (four), the same number of optimizer steps, and the same sweep of eight learning-rate and loss-strength settings on a shared validation split. "Beating" MBPO means two things. First, matching its CHAIR_S within noise on ObjectHal. Second, cutting associate-split and look-alike-split false-yes rates by more than three standard errors relative to it.

The key ablation holds everything fixed except the type composition of the four negatives (all-associate, all-look-alike, uniform, diagnosis-allocated). A second ablation shuffles the type labels, which keeps the same objects but assigns them at random. This checks that the gain comes from the relation structure and not from which words were sampled.

Why not just…?

Why not just train on POPE-style co-occurrence negatives, which already capture the hard case? This is the real threat. If associates drive most hallucinations, typed allocation adds little. POPE's own results show co-occurrence negatives are harder than random ones (MiniGPT-4 drops from 78.86 to 71.37 F1). But the adversarial split was never decomposed, so we do not know how much of that drop is look-alike confusion hidden inside it. Our composition ablation settles the question directly. If all-associate matches diagnosis-allocated on the look-alike split, we drop the allocation claim and keep only the diagnostic.

Why not just corrupt the image, as MBPO and MFPO do? Image corruption teaches the model to depend on pixels, which should help associate errors. It does not create the minimal contrast between a cup and a mug in a clean image that look-alike errors need. MFPO also falls back to global noise in about 10% of cases when segmentation fails. Running both methods on our typed splits will show which error type each one leaves behind. They may well beat us on associates, and that would be an informative result in itself.

Why not just use aggressive modality dropout? Negative to Positive Co-learning with Aggressive Modality Dropout showed that 80% dropout repairs unimodal performance but hurts above that level. Dropout also trains for missing inputs rather than for a language prior that contradicts what the image shows. We include a dropout-trained variant at matched compute as a check.

Why not just ask the judge to reason step by step? MLLM-as-a-Judge found that this can lower accuracy through invented visual details, and The Silent Judge found that judges' explanations hide what actually drove the verdict. Our judge audit includes a reasoning-prompted condition so the question is answered with numbers rather than assumed.

Proposed Experiments

The first decisive check is whether the split is real. On base models, the text-substitution test must show that associate false-yes rates stay close to their real-image level when the image is replaced by the exact object list, while look-alike false-yes rates fall sharply. If the two types behave the same under substitution, the mechanistic claim is false and the rest of the program shrinks to a benchmark contribution.

The second check is the crossed transfer matrix. Train on each negative type, then evaluate on each typed split, on POPE's original splits, on ObjectHal and Object HalBench, and on MME to detect over-refusal. The proposal predicts a clear asymmetry: each type helps its own split more than the other's. The diagnosis-allocated mixture should beat co-occurrence-only negatives at a fixed budget of four.

The third check tests whether repairing associate errors generalizes to prior-driven failures elsewhere. We measure this on VQA-CP v1, where Don't Just Assume; Look and Answer reports SAN at 26.88% and GVQA at 39.23%, and on the counterfactual-conflict tier of C³PO, where the human baseline is 88.64% and Qwen3-Omni scores 43.98%. Associate-trained models should improve there and look-alike-trained models should not.

The last check is the judge audit. It reports, by relation type and by input condition, how often judges catch injected errors and whether their reasons mention them. We will use GPT-4V's reported 0.490 Pearson correlation with human scores on MLLM-as-a-Judge as the reference for how much any open-ended score can be trusted.