Confusion-Aware Diversity Reranking for Multimodal Retrieval-Augmented Answering

When a vision-language model answers a question about an image by reading retrieved documents, the documents that hurt it most are not random junk. They are near misses: the encyclopedia page for a similar-looking bird, or a photo of the neighbouring cathedral. These near misses are plausible enough that the model copies their facts. The standard tool for cleaning up a retrieved list is maximal marginal relevance (MMR), a greedy reranker that picks each next item by trading off similarity to the query against similarity to items already picked. MMR measures "redundancy" with embedding cosine similarity, and in mixed image–text pools that measure is wrong in two ways. Images and texts sit in separate regions of a shared embedding space, the so-called modality gap. As a result, an image and the article describing the same entity look novel relative to each other, while two different but visually similar entities look redundant. MMR therefore spends its diversity budget on the wrong items.

We propose to replace MMR's cosine redundancy term with a small learned confusability score. This score predicts whether adding a candidate next to an already-selected item will lead the answering model to swap one entity's facts for another's. We train it on synthesized look-alike distractors that humans have verified. The claim is that set-level selection driven by confusability, rather than similarity, recovers most of the accuracy lost to "dangerous neighbour" distractors in knowledge-based visual question answering. It should do so without retraining the answering model, and more cheaply than pointwise reranking or generator-side robustness training.

Core Claim

Rethinking Relevance showed in text-only retrieval-augmented generation that semantically similar distractors are toxic: adding 18 of them dropped Llama 2 accuracy from 56.42% to 37.81%. Random documents instead raised accuracy from 18.66% to 25.30%. This result motivates our proposal. It says the quantity a reranker should suppress is not similarity as such, but a specific kind of similarity that creates confusion.

We argue this matters more in the multimodal setting, for two reasons. First, fine-grained entity recognition is where vision-language models are weakest. Second, the geometry of cross-modal embeddings gives cosine similarity no consistent meaning across pairs from different modalities. Learning Robust Visual-Semantic Embeddings found that contrastive training aligns individual pairs while the two modality clusters stay apart. UniIR found that retrieval without instructions returns the wrong modality about 59% of the time. Together these findings mean MMR's redundancy term is miscalibrated in exactly the pools where distractors do the most damage.

Concretely, we keep MMR's greedy structure. At each step we select the unselected candidate $d$ that maximizes

$$\lambda \, r(d, q) - (1-\lambda) \max_{s \in S} C(d, s \mid q),$$

where $q$ is the image-plus-question query, $S$ is the set already selected, $r$ is a relevance score from an existing reranker, $\lambda \in [0,1]$ sets the trade-off, and $C(d,s \mid q)$ is the learned probability that $d$ describes a different entity or fact that the answering model would confuse with $s$ for this query. Two properties separate $C$ from cosine similarity. It is low for a same-entity image–text pair that supports the same answer, so both can be kept as corroborating evidence. It is high for a look-alike entity even when the embedding distance between the two items is moderate.

Relation to Prior Work

The closest prior work falls into three groups.

The first applies MMR to new settings but keeps cosine redundancy. SEDO (Enhancing Vision-Language Models for Global Cultural Understanding) diversifies retrieved images across regions and reports 88% relevance and 79.75% regional diversity on GlobalRG. The retrieved images are the final output there, and no answering model consumes them. DICL (Exploring the Role of Diversity in Example Selection for In-Context Learning) applies MMR to text in-context examples and wins in 17 of 24 configurations. MMR-BERT (Bi et al.) learns to down-weight candidates similar to content the user rejected. We borrow its idea of making the penalty term learned, but our training signal comes from the answering model's confusion rather than from user feedback.

The second group reranks multimodal candidates one item at a time. GMR (Supervised Fine-Tuning or Contrastive Learning?) is a strong reranker built on Qwen2.5-VL that stays stable from top-25 to top-100 candidates. EntRAG (EntRAG) combines entity-level and learned relevance scores. Neither can represent the fact that two individually relevant items become harmful together. That interaction between items is the only thing our method adds.

The third group makes the answering model robust instead of cleaning its input. DAS (Improving Zero-shot Reader) adds an abstention option and reranks answers using $P(\text{query} \mid \text{document})$. Ext2Gen (Aligning Extraction and Generation) trains the answering model with preference optimization to extract evidence before answering, using synthesized noisy contexts, and reaches 0.557 average against 0.473 for the untrained model on Llama3.1-8b. Both are text-only. Ext2Gen is also the most serious competitor, and we treat it that way below.

Two further papers shape the human-verification design. VisIT-Bench shows that human-written, instruction-specific visual descriptions raise GPT-4's instruction-following success from 31% to 91%, so the quality of human input changes what models can do. MiRAGE shows that vision-language models often disagree with humans about whether a visual clip supports a claim. That disagreement is why we do not trust the models themselves to decide whether a synthesized distractor is actually wrong.

Research Program

The work has three parts.

Synthesizing distractors with human verification. For each training query in Encyclopedic VQA (E-VQA) and InfoSeek, we generate candidate distractors automatically. These include nearest-neighbour entities in the image embedding space, sibling entities in the knowledge-base taxonomy, and text passages about the gold entity with one attribute altered by a language model. A model-generated "distractor" is sometimes actually correct: an alias of the gold entity, or a passage that happens to state the true answer. Such items would teach $C$ the opposite of the truth.

Human annotators therefore follow the author–reviewer pattern that Bae et al. found roughly 13 times cheaper than writing from scratch. They see the query, the gold evidence and a candidate, and answer two questions: is the candidate truly about a different entity or fact, and would a person plausibly confuse the two given this image? We target about 5,000 verified pairs. We log annotation time so the human cost can be reported alongside accuracy.

Training the confusability model. $C$ is a light scoring head on top of a frozen multimodal encoder. It is trained on two labels per pair: the human verdict, and a behavioural label equal to the measured drop in the answering model's accuracy when the pair is placed together in its context. The behavioural label keeps $C$ tied to what actually hurts the answering model, and the human label removes false distractors from the training set. To learn how much each label contributes, we train on each alone and on both. Following the finding in Ex-MCR that the within-model modality gap is roughly linear, the encoder features first pass through a per-modality linear correction. The simple fix for the modality gap is therefore included from the start, and our gains cannot be attributed to it.

Multimodal distractor study. Before relying on the method, we re-run the Cuconasu protocol with images. We place the gold evidence among 1 to 18 look-alike distractors, or among random items, and measure the answering model's accuracy. This tests whether the dangerous-neighbour effect, and the surprising benefit of random items, carry over across modalities. The study's value does not depend on how the reranker performs.

Baselines

The first reference point is doing nothing. The answering model receives the top-k items from the first-stage retriever (UniIR CLIP-SF, which reports Recall@5 of 48.9% on M-BEIR in UniIR). We report k from 1 to 20, because simply shrinking k is itself a way to reduce distractors.

The strongest existing pipeline is EntRAG, which reports VQA accuracy of 46.1 on E-VQA and 44.5 on InfoSeek (EntRAG). If its components can be reproduced, we insert our selection step between its reranker and its answer generator, so the comparison changes only the choice of which items enter the context. If not, we cite its numbers as an external reference and compare internally against our own reproduction. Beating it means a statistically significant accuracy gain on both datasets with the same answering model and the same context budget.

The strongest pointwise reranker is GMR-7B, taking its top-k with k tuned on validation data. We found no reported result on Lacuna for GMR on E-VQA or InfoSeek, so we run it ourselves under the same candidate pool and validation budget.

Two ablations isolate the key ingredient. Plain MMR uses cosine redundancy on CLIP embeddings. Gap-corrected MMR uses the same per-modality linear correction as our method but keeps cosine similarity as the redundancy measure. If gap-corrected MMR matches our method, the learned confusability score is unnecessary. A third ablation varies the training labels for $C$: automatic behavioural labels only, human labels only, and both. It prices the human contribution. For every method, $\lambda$ and k get the same validation search of 20 configurations.

Why not just…?

Why not just use a stronger pointwise reranker with a small k? This is a real threat when the gold item is reliably ranked first, because then k=1 removes every distractor. It fails when the retriever is uncertain between look-alikes, which is exactly the fine-grained regime. There, a small k frequently drops the gold, and a large k brings the distractors back. The sweep over k for GMR measures this directly. Our claim holds only if confusion-aware selection beats the best k at every context budget. If it does not, we will report that.

Why not just train the answering model to ignore distractors, as Ext2Gen does? Ext2Gen's gains (Aligning Extraction and Generation) are large and may carry over to images. We will build a multimodal version by applying preference optimization to the answering model, using the same synthesized distractor contexts. If that robust answering model gains nothing further from our reranker, the reranker is unneeded for that model. It would then be justified only as a cheap add-on for frozen or closed answering models, and we will say so. The combination is the most informative cell in the results.

Why not just fix the modality gap and keep ordinary MMR? The linear-gap result from Ex-MCR makes this cheap and plausible, which is why gap-corrected MMR is an explicit ablation. Our hypothesis is that correcting the geometry realigns same-entity image–text pairs but still cannot tell a look-alike entity from a supporting item. Only a signal derived from the answering model's confusion carries that distinction.

Why not skip humans and use the answering model's own accuracy drop as the label? The comparison between automatic-only and combined labels settles this. Given MiRAGE's finding that models are poorly calibrated about whether evidence supports a claim, we expect automatic labels to mislabel aliases and paraphrases as distractors. If they do not, the human step should be dropped.

Proposed Experiments and Decisive Checks

The main evaluation measures VQA accuracy on E-VQA and InfoSeek at fixed context budgets of 3, 5 and 10 items, with two answering models: an open Qwen2.5-VL model and a closed model accessed only through its API. We add a distractor-gap metric: the accuracy with the selected context minus the accuracy when the gold evidence is given alone. Closing this gap is the direct target of the method. We also report the MiRAGE InfoF1 and CiteF1 scores on generated rationales, to check that diversity does not buy accuracy at the cost of weaker grounding. For generalization, we run the selector unchanged on the interleaved image–text subsets of MRB and on GlobalRG retrieval. GlobalRG tests whether confusion-aware selection preserves SEDO-style diversity (79.75% regional diversity) while removing look-alikes.

Three outcomes would decide the proposal. First, the image-based distractor study must reproduce a clear dangerous-neighbour drop. If look-alike images do not hurt more than random ones, the premise fails. Second, confusion-aware MMR must beat both gap-corrected MMR and the best-k GMR at matched budgets. Otherwise, a learned score measuring the interaction between items is not needed. Third, the gain must survive at least partially on top of the Ext2Gen-style robust answering model, or be clearly reported as useful only for frozen models.