Is the Five-Passage Ceiling in Multimodal Retrieval a Training-Data Artifact?

Text-only question answering systems keep improving as they read more retrieved passages. Their multimodal counterparts, which answer questions about images using retrieved encyclopedia text, stop improving after about five. Fusion-in-Decoder, a reader that encodes each retrieved passage separately and then lets the answer generator attend over all of them, gains 6 absolute points on TriviaQA when it moves from 10 to 100 passages (Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering). Two knowledge-based visual question answering systems peak at five retrieved documents and degrade beyond that: MuKA, which retrieves Wikipedia passages paired with entity images (MuKA), and RA-VQA, which converts images to text and trains the retriever and reader jointly (RA-VQA). The usual explanation is that image-plus-text evidence is inherently noisier. We propose a different one. These readers were trained on data with a very narrow scope: templated questions, a single encyclopedic register, and a retriever whose top hit is usually right. They never had to learn to pick a correct passage out of many plausible wrong ones. If so, the ceiling comes from training scope, not from the multimodal setting.

The test is concrete. We train a multimodal reader on evidence sets that are large (up to 100 passages) and deliberately varied in scope. Each set mixes hard same-category distractors, off-register documents drawn from scientific, biomedical, forum and book corpora, and gold facts rewritten into non-encyclopedic registers. We then ask two questions. Does accuracy now keep rising with the number of passages, as it does for text? Can the same frozen reader answer from a swapped-in, non-Wikipedia memory without retraining? A yes to both would mean that external memory can be scaled and swapped for visual reasoning, with dataset scope as the lever. A no would show that the multimodal ceiling is real and architectural.

Core Claim

The central hypothesis has two parts. First, the drop past five passages in multimodal retrieval-augmented question answering is mostly caused by a lack of distractor diversity and register diversity in the reader's training data. Second, a reader trained with scope-diverse evidence sets will improve monotonically with the number of passages $K$ up to roughly 100. It will also stay accurate when the facts it needs appear in registers other than encyclopedia prose.

The three research directions play different roles here. External memory scaling supplies the target behaviour: the FiD scaling curve, and the zero-shot domain adaptation by datastore swapping shown for nearest-neighbour language models in Generalization through Memorization. External knowledge integration supplies the application and the benchmarks: InfoSeek and Encyclopedic-VQA (E-VQA). Dataset scope limitation supplies both the proposed cause and the intervention.

The discourse literature makes the register concern concrete. Beyond The Wall Street Journal finds that how-to guides, interviews and academic prose signal their structure differently from news, and that explicit discourse markers make up only 16.15% of signals in its multi-genre corpus. A reader that has only seen Wikipedia's conventions has little reason to recognize a fact phrased as a forum answer or a methods paragraph.

Research Program

The work has three parts: a diagnosis, an intervention, and a register-shift test.

Diagnosis. We first establish whether the ceiling is a reader problem or a retriever problem. Using MuKA's late-interaction multimodal retriever over its InfoSeek and E-VQA knowledge bases, we measure how often the gold passage appears in the top-$K$ results (recall@$K$) for $K$ from 1 to 100. We also measure reader accuracy at the same $K$. MuKA reports recall@5 of 69.4 on InfoSeek, so about 30% of gold passages fall outside the top five. The file gives no recall at larger $K$, which is why we must measure it.

If recall keeps rising while accuracy falls, the information is being retrieved and then lost by the reader. That is the case our hypothesis addresses. If recall is flat past five, the ceiling belongs to the retriever and the proposal's premise fails. We report that outcome as a result rather than proceeding.

We also apply RA-VQA's two diagnostic ratios. The hit success ratio (HSR) is the share of correct answers that appear in the retrieved passages, and the free success ratio (FSR) is the share that do not. Together they separate answers grounded in the retrieved passages from guesses made from the model's own weights.

Intervention. We train a fusion-style multimodal reader on LLaVA-1.5-7B. Each image-and-passage pair is encoded independently, and the generator attends over all of them, following FiD's design. This avoids quadratic cost in $K$.

The training change is the evidence-set curriculum. For each InfoSeek-Wikidata or E-VQA training question, we sample $K$ uniformly from 5 to 100 and build the evidence set from four sources:

  1. The retriever's own ranked list, which supplies natural hard negatives.
  2. Same-category entity distractors, such as sibling species from iNaturalist or nearby landmarks, together with their images.
  3. Off-register documents drawn from PIN-14M subsets (PIN): PMC, arXiv, Linux-CN forums, Chinese technical markdown, PG19 books and OBELICS web text.
  4. With some probability, a version of the gold passage rewritten into a non-encyclopedic register (how-to, interview, forum reply, academic methods), keeping the answer-bearing fact.

A further fraction of sets contains no gold evidence at all. These teach the reader when to fall back on what it knows from its weights. This matters because of the "tail entity" finding in Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?: end-to-end models sometimes guess obscure entities better than rigid retrieval pipelines do.

The rewriting step uses an open model. The task is restating a known fact, which is easier than the free question generation that SK-VQA found open models fail at: 76% of LLaVA-34B's generated questions were invalid. Even so, we filter rewrites by checking that the answer string is still present and audit a sample by hand.

Register-shift test. We build an evaluation in which the memory changes and the reader does not. The design borrows REVEAL's knowledge-deletion probe, in which re-inserting deleted facts at inference raised accuracy by 6.7% (REVEAL). For held-out InfoSeek-Human and E-VQA test entities, we delete the Wikipedia passage. We then insert the same facts rewritten into held-out registers, meaning registers withheld from the training rewrites, into a datastore padded with PIN documents.

We cross this with question-side shift. Test questions are rewritten through persona styles, in the manner of VQA-GEN, which found that models robust to one kind of shift collapse under joint shifts. We also evaluate on ViQuAE's free-form questions.

Grounding in theory. The curriculum is meant to test a specific idea. Generalization Properties of Retrieval-based Models argues that retrieval works because local neighbourhoods are simple. Its bounds involve a retrieval radius: too small a radius gives high variance, too large gives high bias. In our setting, a larger $K$ plays the role of a larger radius. We hypothesize that the bias term grows quickly only for a reader that cannot reject out-of-neighbourhood evidence. The curriculum aims to reduce that bias penalty without shrinking the neighbourhood.

We do not claim a new bound. The prediction is empirical: where the accuracy-versus-$K$ curve peaks should move rightward as distractor diversity in training grows. We test this by training the reader at several levels of curriculum diversity.

Baselines

The naive reference points are two. The first is the no-retrieval model: LLaVA-1.5-7B answering from its weights alone. The closest reported analogue is PaLI-X at 10.8% on InfoSeek-Human (Hu et al.). The second is the same reader fine-tuned the standard way on top-5 evidence. This reproduces the regime in which MuKA and RA-VQA report their peak.

The strongest existing methods are MuKA and EntRAG. MuKA reports 42.5% accuracy on InfoSeek and 63.1% on E-VQA. EntRAG adds entity binding and reranking before generation and reports 44.5 on InfoSeek and 46.1 on E-VQA (EntRAG). EntRAG's 44.5 on InfoSeek is the strongest reported number in the file.

The two papers' E-VQA numbers cannot both be "best" under one protocol, since EntRAG claims to surpass prior work with 46.1 while MuKA reports 63.1. We will therefore re-run both under a single fixed split and knowledge base before comparing. EntRAG is the most dangerous competitor because it attacks the problem from the other side. It makes the top few passages right, through better entity linking, instead of making the reader robust to many wrong ones.

The SK-VQA-trained reader is the strongest scope-diversity competitor. It is the same backbone fine-tuned on SK-VQA's 2M synthetic question-answer pairs, which lifted ViQuAE zero-shot accuracy from 40.9% to 78.0%. The key ablation is our fusion reader trained with ordinary top-5 evidence instead of the curriculum. It isolates whether any scaling gain comes from the curriculum or simply from the independent-encoding architecture.

Fairness controls. All readers share the backbone, retriever, knowledge base, number of training tokens and hyperparameter search budget. The SK-VQA baseline receives the same token budget as our curriculum.

What counts as success. Beating the baselines means three things. Accuracy at the best $K$ must exceed both EntRAG and MuKA re-run under the shared protocol. Accuracy must not decrease from $K=5$ to $K=50$. The register-shift drop must be significantly smaller than that of the SK-VQA reader.

Why not just…?

Why not just fix entity linking and stay at five passages? This is the most serious alternative. The jump from 18.2% to 45.6% when an oracle entity is supplied (Hu et al.) says linking is where most errors originate. If EntRAG-style reranking at $K=5$ matches our reader at large $K$, the scaling story has no practical value. The diagnosis settles this. If recall@50 is well above recall@5 even after reranking, a reader that can use large $K$ has headroom that linking alone cannot reach. We will also combine the two, putting a reranker in front of our reader, and report whether the gains add.

Why not just fine-tune on SK-VQA's diverse synthetic data? SK-VQA already shows that scope diversity in training helps, and that it improves resistance to distractors in a simulated 10-passage setting. Our proposal could turn out to be a re-discovery of that result. The differences are that SK-VQA diversifies questions and attaches a single prepended context, while we diversify evidence sets up to 100 passages and evidence registers. The experiment distinguishes the two by plotting accuracy against $K$ beyond 10 and by the register-swapped-memory test. If the SK-VQA reader matches ours on both, our claim is wrong and we will say so.

Why not just caption the image and use a text reader, which already scales? RA-VQA does exactly this, yet still peaks at five passages. That suggests the problem is not specific to visual inputs. It is also why we treat training scope, rather than modality, as the suspected cause. MuKA's move from text-only to multimodal retrieval raised recall@5 from 60.1 to 69.4, so dropping images costs retrieval quality. We include a captioned-image FiD reader trained with our curriculum as a control. It shows whether the curriculum effect holds without image evidence.

Why not just use a larger model? REVEAL at 1.4–2.1B parameters outperforms Flamingo-80B on MSCOCO captioning. PaLI-X manages only 10.8% on InfoSeek-Human. Neither result suggests that more parameters substitute for evidence. Scale also does not address swapping the memory to a different register.

Proposed Experiments

Decisive check 1: the scaling curve. We plot accuracy against $K \in \{1, 5, 10, 20, 50, 100\}$ on InfoSeek and E-VQA for the standard-trained reader, the SK-VQA reader, the architecture-only ablation and the curriculum reader, alongside retriever recall@$K$. The prediction is that only the curriculum reader tracks recall upward past five. HSR and FSR should show its extra correct answers to be grounded in the retrieved passages.

Decisive check 2: register-swapped memory. We measure accuracy on deleted-and-rewritten entities across held-out registers, crossed with persona-rewritten questions and ViQuAE. The comparison is against the same entities with the original Wikipedia passages restored. The quantity of interest is how much accuracy each reader loses relative to that restored condition.

Decisive check 3: the curriculum dose-response. We measure the peak $K$ as a function of the share of off-register and same-category distractors in training. This tests the local-regularity reading described above.

A secondary analysis asks whether the multimodal analogue of knowledge corpus error exists. That paper found that rewriting gold passages into cleaner forms raised reader accuracy by 10–13% on text QA. We test whether rewritten gold passages help or hurt accuracy, separately for correctly linked and mislinked queries.

Outcomes and their meaning. A positive result would be a monotone scaling curve together with a small register gap. A negative result would be a flat scaling curve despite rising recall. The negative result would still be useful: it would locate the ceiling in the multimodal fusion itself and redirect effort toward architecture.