When the Judge Cannot See the Image: Self-Preference in Memories of Visual Summaries

Many vision-language systems no longer reason over pixels directly. A small captioning model first writes a text summary of the image, often tailored to the question being asked, and a separate language model reads that summary, sometimes alongside many other stored summaries, and produces the answer. This split works well. A 2B captioner feeding a large text model can rival far larger end-to-end systems (Prism). It also creates a weakness that nobody has measured. The text model must decide which summary to believe, but it has no access to the image that would let it check the summary. Language models are known to favor text that reads like their own writing. In this pipeline, that habit would push the reasoner toward summaries from captioners of its own model family, including summaries that describe objects that are not in the image.

We propose to test one specific hypothesis, which we call verifiability gating. Self-preference is suppressed when the judge can check a claim against its own knowledge, and it returns when the claim can only be verified by looking at the image. This would reconcile two findings that currently sit apart. In text retrieval, factual correctness overrides self-preference (LLMs are Biased Evaluators But Not Biased for Fact-Centric RAG). In open-ended summarization, models recognize and favor their own outputs (LLM Evaluators Recognize and Favor Their Own Generations). If the hypothesis holds, it predicts a concrete failure. As an external store of visual summaries grows, a self-selecting reasoner should increasingly pick its own family's hallucinations, so accuracy stops improving even though correct summaries are available in the store.

Core Claim

Fact-centric text retrieval shows no self-preference (Chen et al.), and we think the reason is that the judge can test each passage against what it already knows. Visual summaries break this condition. A claim such as "the sedan is red" or "there is a napkin on the table" is true or false only with respect to pixels the text reasoner never sees. Our claim is that the size of self-preference depends on the type of claim, in three regimes. When the claim is consistent with world knowledge, self-preference should be near zero. When the claim is only checkable against the image and neutral with respect to world knowledge, self-preference should be large. When the image contradicts world knowledge, the reasoner's knowledge, not authorship, should drive its choice. That third case is the language-hallucination regime of HallusionBench.

The contribution has three parts. The first is a measurement design that separates authorship from correctness and from world-knowledge consistency. The second is a three-way attribution of pipeline errors. A wrong answer can come from a captioner misreading the image, from the reasoner selecting the wrong stored summary, or from the reasoner overriding correct evidence with what it already believes. The third is a scaling curve that shows whether biased selection caps the benefit of adding more summaries to memory.

Where This Sits in Prior Work

Two papers supply the pipeline. PromptCap trains a question-guided captioner and shows that relevant detail matters more than completeness. Prism separates perception from reasoning and evaluates each stage. Neither asks what happens when the reasoner must choose among competing summaries, or whether its choice depends on who wrote them. RA-VQA converts images to text and retrieves outside passages. Its image-to-text step is generic rather than question-conditioned, it reports retrieval of distracting passages as a weakness, and it never examines the authorship of the stored text.

The self-preference literature motivates the hypothesis and also constrains the design. Panickssery et al. show that self-recognition drives self-preference in text summarization: GPT-4 identifies its own summary 73.5% of the time, and fine-tuning lifts other models above 90%. Self-Preference Bias in LLM-as-a-Judge attributes the bias to lower perplexity on the judge's own text. Chen et al. is the result we must explain rather than contradict. When models had no prior knowledge of the answer, they still picked the factual passage over their own non-factual one 65–70% of the time. Our hypothesis predicts that this rate falls substantially when the fact is visual. Are LLM Evaluators Really Narcissists? and Blind to the Human Touch warn that position, length, and word overlap can masquerade as self-preference. These confounds dictate the controls below.

The hallucination literature supplies the ground truth and the error categories. POPE provides object-existence labels with adversarial distractors that commonly co-occur with real objects. It also documents near-99% "yes" rates in some open models. HallusionBench provides original/edited image pairs. On those pairs, GPT-4V drops from about 75–80% accuracy on originals to about 38% on edits. These pairs are exactly what is needed to construct the "image contradicts world knowledge" regime. VCBench attributes about 62% of Gemini 2.0 Flash errors to perception. Our attribution asks how much of that "perception" error is actually a selection error made downstream.

Research Program

Building the summary memory. We build a memory of visual summaries for images from POPE (COCO), A-OKVQA, OK-VQA, and HallusionBench. For each image and question, captioners from at least three model families write summaries in both generic and question-conditioned modes, with several sampled variants each. The captioners are LLaVA-based PrismCaptioner (Llama lineage), a Qwen-VL captioner (Qwen lineage), and an InternVL captioner (InternLM lineage). Text-only reasoners from the same three families, Llama-3, Qwen2.5, and InternLM, then read retrieved summaries and either rank them or answer from them. This crossing makes every reasoner a "sibling" of exactly one captioner, the lineage effect described by PARIKSHA.

Labeling each summary. Each atomic claim in each summary is labeled as correct or hallucinated against COCO annotations, POPE probes, and HallusionBench edits. Each claim is also tagged as world-knowledge-consistent, image-only, or knowledge-contradicting. Where possible we tag claims automatically, by checking whether a text-only model answers the probe correctly without the image. We validate a sample of these tags by hand.

Measuring self-preference. We measure two quantities. The first is matched self-preference: the rate at which the reasoner picks its sibling's summary over a non-sibling summary when both are equally correct and length-matched, minus 0.5. The second is harmful self-preference: the rate at which it picks a hallucinated sibling summary over a correct non-sibling summary. Both are computed separately for each claim regime and for each captioning mode.

Establishing causation. Following Panickssery et al., we use three manipulations. We swap authorship labels to change perceived identity. We paraphrase summaries with a neutral fourth-family model to strip the stylistic signature. We record reasoner perplexity on each summary to test the mechanism proposed in Self-Preference Bias in LLM-as-a-Judge.

Attributing errors. Every wrong final answer is assigned to one of three causes. It is a perception error if no stored summary contained the correct evidence. It is a selection error if correct evidence was in memory but the reasoner preferred a summary that lacked it. It is a knowledge-override error if the reasoner chose the correct summary and still answered from prior belief. This extends the visual-illusion versus language-hallucination split of HallusionBench to retrieval pipelines.

Scaling the memory. We vary the number of candidate summaries per image, N, from 2 to 16, while keeping the mix of families balanced. For each N we record downstream accuracy, the fraction of selections that go to the sibling captioner, and the gap to oracle selection.

Question-conditioning enters as a secondary factor. A question about a co-occurring object such as "the napkin" may induce the captioner to describe something absent. If question-conditioned summaries carry both more useful evidence and more hallucinations of this kind, the selection step matters even more.

Baselines

The naive reference is a single generic caption from a single captioner, passed to the reasoner, with no memory and no selection. This is the information-bottleneck setting that PromptCap improves on by 3.8% on OK-VQA and 9.2% on VQAv2.

The strongest existing method for question-guided captioning is PromptCap, which reports 60.4% accuracy on OK-VQA and 59.6% on A-OKVQA. We run it as a single-summary pipeline with the same reasoners.

For decoupled perception and reasoning we compare against Prism. It reports 45.9% on MMStar with PrismCaptioner-7B plus Llama-3-70B. We also add MMStar as a secondary benchmark so this comparison is direct.

For retrieval-based VQA we cite RA-VQA at a 54.48 VQA score on OK-VQA as context. Our memory holds generated summaries rather than Wikipedia passages, so we do not claim a like-for-like comparison.

The baseline most important to our question is family-blind aggregation. Here every retrieved summary is concatenated, Fusion-in-Decoder style (FiD), and the reasoner answers without ever ranking the summaries. It is the strongest competitor because it sidesteps selection entirely. If it matches oracle selection, then our bias is real but does not matter in practice.

We also include a vision-grounded judge: a VLM from a fourth family that ranks the summaries while looking at the image.

The ablation that isolates our key ingredient is neutral paraphrasing of every summary before it is stored.

All conditions share the same images, the same retrieved candidate pools, the same reasoner checkpoints, and greedy decoding. Each prompt is tuned on a held-out 200-example split with an equal budget, and candidate order is randomized and counterbalanced. Our selection-aware design counts as an improvement only if it beats family-blind aggregation on A-OKVQA and POPE-derived accuracy at N≥8 with non-overlapping bootstrap intervals. It must also cut harmful self-preference on image-only claims by at least half relative to the unmodified self-selecting reasoner.

Why not just…?

Why not just paraphrase every summary through a neutral model? This is the most serious threat, and we treat it as a hypothesis to be tested rather than an objection to be answered. If paraphrasing removes harmful self-preference without adding new hallucinations or dropping question-relevant detail, the practical recommendation becomes simply "paraphrase before storing." The mechanistic finding about verifiability gating would still stand. We will measure claim-level hallucination and the loss of detail introduced by paraphrasing. The paraphrase-only ablation tells us whether the elaborate design is needed. Panickssery et al.'s label-swap result, where telling a model a competitor's text was its own inverted its preference, suggests that perceived identity may survive paraphrasing. That is why we run both manipulations.

Why not just let a vision-language model judge the summaries against the image? Checking against the image directly addresses the verifiability gap, and we include it as a baseline. The concern is that such a judge has its own failures. GPT-4V reaches only 31.42% question-pair accuracy on HallusionBench, and open models show near-99% "yes" rates on POPE. An image-aware judge may therefore trade self-preference for language-prior hallucination. The error attribution reports which of the two it introduces, and at what added compute.

Why not just aggregate all summaries and skip selection? Family-blind aggregation is the baseline described above. Its weakness is that length grows with N, and the concatenated text still contains sibling-styled passages that may dominate attention. We test this by looking for a sibling-family bias in which evidence the answer follows, even without explicit ranking.

Isn't this just position or length bias? The design includes length-matched pairs, counterbalanced order, and perplexity controls, following the critique in Are LLM Evaluators Really Narcissists?. If the effect does not survive these controls, we report it as an artifact.

Proposed Experiments and Decisive Checks

The first decisive check is the regime interaction. Harmful self-preference should be near zero for world-knowledge-consistent claims, replicating Chen et al. in the visual setting, and clearly positive for image-only claims on POPE adversarial and A-OKVQA. A flat profile across regimes falsifies verifiability gating.

The second check concerns knowledge-contradicting claims from the HallusionBench edited images. There, selection should track the reasoner's prior knowledge regardless of which captioner wrote the summary. This would show that authorship and prior knowledge are separable forces.

The third check is the scaling curve. If a self-selecting reasoner's sibling share rises with N while accuracy plateaus below oracle selection, biased selection is capping the benefit of a larger memory. If family-blind aggregation or paraphrasing restores the trend toward oracle accuracy, the cause is localized.

Finally, rerunning the VCBench error analysis with our three-way attribution will show what fraction of the reported 62% "perception" errors are in fact selection errors. These would be cases where a correct description of the image existed in memory and the reasoner preferred its sibling's wrong one.