Image-Complementary Utility for Multimodal Retrieval

When a vision-language model answers a question about a photo with help from retrieved web text, the retriever usually picks passages that are about the same topic as the question. Recent text-only work shows that a better selection signal is utility: how much a passage raises the model's probability of producing the correct answer. In the multimodal setting this signal has a blind spot. A passage can raise the probability of the right answer because it lets the model skip looking at the image. A page about a look-alike product, or about "the tallest tower in Paris," fully answers "how tall is this building?" whether or not the photo shows that building. A reranker trained to imitate plain utility learns to prefer these shortcut passages. At test time, when the top passage describes a similar-looking but different entity, the model adopts it and hallucinates.

We propose to measure and train for a narrower quantity: utility that exists only because of the real image. Concretely, we score a passage by how much more it helps when the true image is present than when the image is replaced by a counterfactual image, such as a look-alike or blank. This is a difference-in-differences of answer log-likelihoods. It is computed offline with the generator, then distilled into a small multimodal reranker, so inference cost is a single reranker pass. The bet is specific. Plain multimodal utility and our image-complementary utility should look similar on standard benchmarks. Under look-alike distractors, plain utility should fail and image-complementary utility should not.

Core Claim

Let $q$ be the question, $I$ the image, $D$ a candidate passage, $y$ the gold answer, and $P_\theta$ the generator. The direct port of text utility, as in InfoGain-RAG, would score

$$G(D; I) = \log P_\theta(y \mid q, I, D) - \log P_\theta(y \mid q, I).$$

We instead score

$$U(D) = G(D; I) - \mathbb{E}_{\tilde I \sim \mathcal{C}(I)}\, G(D; \tilde I),$$

where $\mathcal{C}(I)$ is a small set of counterfactual images. Each counterfactual is either a nearest-neighbor image of a different entity retrieved by image embedding, or a blank image.

The three cases separate cleanly. A passage that helps equally with or without the true image has $U \approx 0$; it is a text shortcut. A passage that helps only when combined with what the photo shows has $U > 0$; it is a bridge, for example facts about the entity the image identifies. A passage that helps more under the wrong image has $U < 0$; it describes a look-alike, or it conflicts with the visual evidence.

The claim is that training a reranker on $U$ rather than $G$ reduces retrieval-induced hallucination when look-alike distractors are present, without the coverage collapse of post-hoc verification.

Relation to Prior Work

Two strands motivate the gap.

On the retrieval side, Dual Preference Alignment names the "unaligned knowledge" case, where high-similarity passages mislead an otherwise correct model. Counterfactual Reasoning for RAG describes a "correlation trap," where correlated but non-decisive evidence wins out. Both are text-only, and neither asks what the passage does relative to another modality.

On the vision side, Robust Multimodal LLMs Against Modality Conflict shows hallucination rates above 40% for GPT-4o and Qwen2-VL when text premises contradict the image. Villa et al. show that correct answers often come from guessing without visual grounding. Together these say retrieved text can override the image, and that accuracy alone hides it. That is exactly the failure a plain-utility label would reward.

Three methods supply our technique. InfoGain-RAG defines document information gain as a likelihood shift and distills it into a 355M reranker. Relevance to Utility (R2U) distills log-likelihood deltas into a rewriter. GRO-RAG approximates leave-one-out loss with gradients. We reuse the distillation recipe and change the label: our label conditions on a counterfactual image, which none of these can express because none has an image. GRO-RAG's reported cost of about 882ms per sample on LLaVA-scale models also constrains us. It is why we compute labels offline and never run the generator inside the reranker at test time.

Closest at the intersection are four multimodal systems, all of which rank with signals that cannot see the shortcut. Adaptive Termination via Cross-Modal Evidence Grounding scores evidence with CLIP and BGE cosine agreement; its authors note this measures relatedness, not entailment or utility. Multimodal Reranking for Knowledge-Intensive VQA trains a PaLI-3B cross-encoder with answer-string-match labels. Active Retrieval Augmentation reranks by caption similarity. CRUISE filters after generation. A look-alike passage contains the answer string, is semantically close, and is consistent with a plausible caption, so all four signals score it highly.

Retrieve-then-compare also uses retrieved look-alike images, but as decoding-time "hallucination templates" to subtract logits. It does not select knowledge, and it pays about 0.43x latency per reference image. We borrow its look-alike retrieval for a different purpose: building counterfactuals at labeling time.

Research Program

Labeling. For each training question in CRAG-MM's single-turn split (CRAG-MM) and in OK-VQA, we retrieve the top 50 passages with the benchmark's standard retriever. For each passage, we compute $G$ under the true image and under three counterfactuals: two look-alike images of different entities found by image-embedding search over the benchmark's image pool, plus one blank image. We use an open generator that exposes log-probabilities (LLaVA-1.5 and Qwen2-VL-7B, run separately). We apply InfoGain-RAG's length smoothing to answers that span several tokens. Labeling costs four times the plain-utility cost, but it is a one-time offline expense.

Reranker. A 2B-scale vision-language cross-encoder reads image, question, and passage. It is trained with InfoGain-RAG's combination of a classification loss (sign of $U$) and a margin ranking loss on $U$. At inference it reranks the top 50 to a top 3, and passages with predicted $U < 0$ are dropped. Because a frozen generator then reads only the selected passages, the reranker is the sole intervention.

Analysis. We will hand-annotate 300 passages that have high $G$ but low $U$. The goal is to measure what fraction are wrong-entity or premise-conflicting. This directly tests whether the shortcut population exists and is what $U$ removes.

Baselines

Naive multimodal RAG. The generator reads the top-3 passages by retriever similarity. CRAG-MM reports roughly 32% single-turn truthfulness for straightforward RAG, and GPT-5 and Gemini-2.5-Flash score around the same (CRAG-MM). We rerun naive RAG with our generators so every system shares a retriever and passage pool.

Plain multimodal information gain (MM-DIG). This is the reranker trained identically to ours, with the same architecture, data, steps, and hyperparameter search budget, but on label $G(D; I)$ alone. It is the ablation that isolates the counterfactual-image term, and it is the strongest competitor on the core question. For a multimodal version we found no reported result on Lacuna. For context only, the text-only original reports +17.9% EM over naive RAG on NaturalQA (InfoGain-RAG). Beating it means equal or better accuracy on the standard splits plus a significant reduction in hallucination on the look-alike stress split.

Distant-supervision cross-encoder. This follows Multimodal Reranking for Knowledge-Intensive VQA, retrained with our backbone and budget. That paper reports 52.6% OK-VQA accuracy with its PaLI-3B reranker, against an oracle ceiling of 64.4%.

Embedding-agreement filter. This is CMEGS-style CLIP/BGE image-passage agreement thresholding from Adaptive Termination. It is a cheap competitor that also tries to reject passages inconsistent with the image.

Post-hoc verification. This is CRUISE, whose reported CRAG-MM numbers are 2.88% hallucination, 82.69% missing rate, and truthfulness 11.54, against −34.62 for its unverified agent. These come from a different leaderboard pipeline than CRAG-MM's 32% figure, so we compare only against our own reruns using the same scoring script.

Decoding-time counterfactual. Last, we contrast the generator's logits with true versus look-alike images at every token, applied to naive-RAG passages. This tests whether the counterfactual helps more at selection time or at decoding time.

Why not just…?

Why not just port InfoGain-RAG to images and score with $G$? This is the real threat. If shortcut passages are rare in practice, $G$ and $U$ will rank nearly identically, and $U$ only adds labeling cost. The experiment settles it in three ways. Rank correlation between $G$ and $U$ labels on held-out data will show whether the two signals differ at all. The 300-passage annotation will show whether high-$G$, low-$U$ passages are wrong-entity. The stress split will show whether the $G$-trained reranker actually admits them. If all three come back null, we report that plain MM-DIG suffices.

Why not just filter passages by CLIP image–text agreement? Embedding agreement rewards look-alikes by construction: a page about a similar product is visually and semantically close to the photo. The Adaptive Termination authors themselves concede the score is not entailment. The look-alike stress split is designed to make this baseline fail, and if it does not, our premise is wrong.

Why not just verify after generation? CRUISE shows that verification drives hallucination down only by abstaining on more than four in five questions. We report missing rate alongside hallucination, and we expect to beat CRUISE on truthfulness, which penalizes both errors and excessive abstention. Verification can also be stacked on our reranker, so we test that combination as well.

Why not just apply the counterfactual image at decoding time? This avoids training, but it doubles per-token cost, in line with the overheads reported for RITUAL and Retrieve-then-compare. It also cannot remove a bad passage that is already in the context; it can only damp tokens that do not depend on the image. Our decoding-time baseline measures this directly, and a tie at lower cost would still favor the reranker.

Proposed Experiments

The main evaluations are CRAG-MM single-turn, scored with truthfulness, hallucination rate, and missing rate from the benchmark's scorer; OK-VQA accuracy; and Dyn-VQA F1-recall as a transfer test without retraining. On Dyn-VQA, OmniSearch with GPT-4V reports 50.03% against 30.25% for base GPT-4V (Dyn-VQA), though with a different generator than ours.

The decisive experiment is a look-alike stress split. For 1,000 CRAG-MM and OK-VQA test questions, we insert into the candidate pool passages about the look-alike entity whose image is nearest by embedding to the true one. These passages are held out from the entities used to build training counterfactuals, so the reranker cannot memorize them. We also measure an image-swap test. When the true image is replaced by the look-alike while passages are re-retrieved, the system's answer should change. A system that answers identically is answering from text alone, which is the hidden-grounding failure Villa et al. describe.

The idea is supported if three things hold together on both generators. On standard splits, the $U$ reranker matches or beats the plain-$G$ reranker. On the stress split, it cuts hallucination substantially relative to plain $G$ and to the embedding filter. And it achieves truthfulness above CRUISE-style verification, rerun on our setup, with a missing rate far below 82.69%.

The idea is refuted if the plain-$G$ reranker is equally robust on the stress split. As secondary checks, we vary the counterfactual type (look-alike only, blank only, both) to learn which carries the signal. We also confirm on POPE that closed-book behavior is unchanged, as a sanity check that the reranker does not alter the generator.