Vision-language models describe objects that are not in the picture. They mostly do this because their language half has learned which objects tend to appear together: kitchens have ovens, and streets have cars. The usual fix is to fine-tune the model on preference pairs, meaning a faithful description it should prefer and a hallucinated one it should reject. Building good pairs is the bottleneck. Asking GPT-4V to judge is slow and the judge itself hallucinates. Sampling many answers from the model and scoring them with its own probabilities is fast, but the samples are near-copies of each other. They therefore rarely contain the specific, plausible mistakes the model is prone to.
We propose to source the hard mistakes from retrieval and to judge them with the model's own hidden states. For each training image, we retrieve the captions of visually similar images. These captions contain exactly the co-occurring objects, attributes, and relations that drive hallucination. We splice each retrieved claim into the model's own response. We then let the model's middle-layer representations decide whether the image actually supports the claim, because prior work shows those representations often "know" the answer even when the model would say yes anyway. The result is a diverse, self-labeled set of preference pairs that targets the model's own failure modes. It needs no external judge and no human span annotation.
Core Claim
Two weaknesses of current self-supervised hallucination training can be fixed together. The first is that the candidate pool is homogeneous. The second is that the labels are noisy. Retrieval supplies varied, realistic counterfactual claims. The gap between what a model internally represents and what it says supplies labels more reliable than the model's verbal answers. Preference data built this way should cut hallucination on attributes and relations as well as object existence, at matched data size and compute.
The claim rests on three findings and one weakness.
The first finding concerns where hallucinations come from. POPE showed that object hallucination tracks how often objects co-occur in the training data, and that models have a strong "yes" bias when probed. Nearest-neighbor captions are therefore a natural generator of the right negatives.
The second finding concerns what the model knows internally. Bridging the Knowledge-Prediction Gap found that in text models, the directions encoding correct answers and the directions encoding the model's actual outputs drift toward orthogonality in late layers. SHARP found that faithful and hallucinated states in vision-language models are linearly separable, with the separation peaking in middle layers. Both suggest that intermediate states carry more truth than the emitted text.
The third finding supplies the labeling method. ContextualLens takes middle-layer embeddings of answer tokens and compares them to image-patch embeddings. On attribute hallucination it reaches 0.911 mean average precision, against 0.830 for the token-level "logit lens," a method that reads hidden states by projecting them onto the vocabulary. ContextualLens is only a post-hoc detector. We reuse its technique as a labeler for training data.
The weakness is in IRIS. IRIS selects preference pairs from the model's own samples using "rectified visual guidance": a candidate is penalized when it is likely without the image but unlikely with it. IRIS reports strong object-existence gains, but its gains on compositional questions (MMHal-Bench) were modest. Its pool of $K$ samples inherits the narrowness of the model's sampling distribution.
Research Program
The pipeline has four stages.
Retrieval. For each image in a fixed training set, we take the same 5.7k RLHF-V images IRIS used. We retrieve the top neighbors by image-embedding similarity from a large caption corpus: MS-COCO captions plus the phrase-level spans of Grounded Visual Chat. We parse their captions into atomic claims: an object, an attribute bound to an object, or a relation between two objects. This choice departs from RADA, which also uses retrieval to break the "diversity crisis" of few-shot synthetic data. RADA is text-only and treats retrieved passages as topical seeds. Here, retrieved content serves as candidate counterfactuals about a specific image.
Splicing. The model first writes its own description. Each retrieved claim not already present is inserted by a local edit at the most compatible position. To keep the data close to the model's own distribution, as IRIS argues it must be, we discard edits whose likelihood under the model falls below a threshold. In other words, we keep only claims the model itself finds plausible.
Internal labeling. For each claim $c$, we compute an evidence score:
$$e(c, I) = \max_{p \in P(I)} \cos\big(h_\ell(c),\, v_\ell(p)\big)$$
Here $h_\ell(c)$ is the middle-layer contextual embedding of the claim's tokens, $v_\ell(p)$ is the embedding of image patch $p$ at the same layer $\ell$, and $P(I)$ is the set of patches of image $I$. Layer $\ell$ is chosen once, on a held-out annotated split, as the depth where faithful and hallucinated claims separate best. Claims with high $e$ that the model's description omitted become additions to the preferred response. This mirrors the omitted-object recovery of PatchGate, extended from single-word nouns to attributes and relations. Claims with low $e$ become the edits that define the rejected response. Claims in between are dropped.
Counting claims are excluded by design. ContextualLens reports that number tokens fail under patch matching, and we do not claim to fix that.
Training and measurement. The pairs feed standard DPO (direct preference optimization, which trains the model to raise the likelihood of the preferred response relative to the rejected one). We measure the diversity of each method's pool with DCScore on claim-text embeddings. DCScore is a classification-based diversity metric that is cheaper than eigenvalue-based scores. Its authors note it is unimodal, which suits claim text.
The central analysis is not the final benchmark score. It is whether the internal evidence score labels claims better than the model's own spoken yes/no answer. That is a direct test of the internal-versus-external gap in the multimodal setting.
Baselines
Do nothing. The naive reference point is untuned LLaVA-1.5-7B, the backbone shared by IRIS, Synth-Align, PatchGate, and SHARP. On Object-HalBench, IRIS reports a sentence-level CHAIR of 54.70 for this model (the share of descriptions containing at least one hallucinated object). On MMHal-Bench, Synth-Align reports a score of 2.36 and a hallucination rate of 51.0%.
IRIS. This is the strongest competitor on the precise question, because it is the existing self-judged, judge-free preference method. IRIS reports that two rounds of training cut Object-HalBench CHAIRs from 54.70 to 8.66 on LLaVA-1.5-7B. We rerun it on the same 5.7k images, with the same number of pairs, DPO hyperparameters, and GPU-hours, so that only the candidate source and labeler differ. Beating it means matching it on Object-HalBench and clearly exceeding it on the AMBER attribute and relation dimensions and on MMHal-Bench, where IRIS's gains were modest. No reported MMHal-Bench number for IRIS was found on Lacuna, so we report our rerun.
Synth-Align. This represents the external-judge route. Synth-Align builds pairs with GPT-4 prompts, diffusion-generated images, and an external reward model. It reports MMHal-Bench going from 2.36 to 3.49, a hallucination rate falling from 51.0% to 25.0%, and POPE accuracy of 87.6%, all on LLaVA-1.5-7B. We cite these numbers and do not rerun it, because its data budget differs. The claim is parity or better without any closed model.
Training-free decoders. These test whether training is needed at all. PatchGate reports POPE F1 of 89.6% and an AMBER CHAIR that drops from 7.5 to 6.6. SHARP reports CHAIRs falling from 52.8 to 34.8 on LLaVA-v1.5.
Ablations. Three ablations isolate the two key ingredients.
The first replaces retrieved claims with IRIS-style samples at a larger pool size $K$, with the internal labeler held fixed. This isolates the effect of retrieval on diversity.
The second keeps retrieval but labels every unmatched retrieved claim as a hallucination by construction, in the manner of Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data. That paper's T5-inserted distractors were non-hallucinatory 34% of the time, so this ablation isolates the value of the internal labeler.
The third replaces the internal labeler with the model's own verbal answer to "Is there a [claim] in the image?". This isolates the internal-versus-external gap itself.
Why not just…?
Why not just sample more candidates at higher temperature? Raising temperature does increase diversity. DCScore tracks temperature with Spearman correlation above 0.9. But CorrSynth shows that unconstrained sampling drifts into incoherent text. It also shows that structured sampling preserves quality better: MAUVE, a measure of how close generated text is to real text, reaches 94.4 versus 87.1 for standard few-shot sampling on AG News. More importantly, hotter samples are random departures from the model's typical answer, not the co-occurrence errors POPE identifies. The first ablation settles this question at matched pool size and labeler.
Why not just label retrieved claims with ground-truth annotations? This is a real threat on COCO, where object boxes exist. If annotation labels perform as well, the internal labeler is unnecessary there. Our answer is scope. COCO annotations cover 80 categories and no attributes or relations, and the method is meant for unannotated images. We run annotation labels on the COCO subset as an upper bound. The internal labeler is justified only if it approaches that bound on objects and extends beyond it on attribute and relation claims, which we verify against M-HalDetect spans.
Why not just ask GPT-4V to judge? IRIS reports that the GPT-4V-judged RLAIF-V pipeline took 66 hours, against 1.5 hours for internal sifting. Studies of model-as-judge evaluation, such as Chen et al., report judges that invent errors and misread layouts. We still report GPT-4V labels on a 1k-claim audit set as a reference, not as the training signal.
Why not just decode with PatchGate or SHARP at inference time? Both are cheap and effective on object existence. PatchGate, however, is restricted to single-word WordNet nouns, and SHARP's steering vectors are fixed per dataset. If they match the trained model on AMBER relations and MMHal-Bench, the training stage is not worth its cost. We also test whether the two approaches stack.
Proposed Experiments
The first decisive check runs before any training. On the COCO subset with object annotations and on M-HalDetect spans, we compare three labelers of spliced claims by AUROC (area under the curve of true-positive against false-positive rate, so 0.5 is chance). The three labelers are the internal evidence score, the model's verbal yes/no answer, and IRIS's visual-guidance score. The corrupted-grounding paper reports human span agreement of about 86%, which sets a ceiling. If the internal score does not beat the verbal answer, the know-tell premise fails in this setting and the program stops there.
The second check measures the diversity of each method's pool with DCScore and the fraction of claims that are attributes or relations. We expect the retrieved pool to be measurably more diverse than IRIS's pool at equal size.
The third check is the main comparison. All trained variants are evaluated on Object-HalBench (CHAIRs), AMBER (CHAIR, coverage, and the existence, attribute, and relation scores), POPE in its adversarial split (F1), MMHal-Bench (score and hallucination rate), and MME (perception total) as a guard against capability loss.
The prediction is specific. Gains over IRIS should concentrate in AMBER's attribute and relation dimensions and in MMHal-Bench, with parity on Object-HalBench. If the gains instead appear only on object existence, retrieval is doing nothing beyond what more sampling achieves.
References
- Evaluating Object Hallucination in Large Vision-Language Models
- Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- Beyond Logit Lens: Contextual Embeddings for Robust Hallucination Detection & Grounding in VLMs
- IRIS: Implicit Reward-Guided Internal Sifting for Mitigating Multimodal Hallucination
- Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks
- PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Models
- Measuring Diversity in Synthetic Datasets
- Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
- Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data
- CorrSynth - A Correlated Sampling Method for Diverse Dataset Generation from LLMs
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark