Fictional Worlds for Teaching Vision-Language Models to Read Retrieved Evidence

Vision-language models can describe a photo of a bridge but cannot say who designed it, because the pixels do not contain the answer. The standard fix is retrieval: look the entity up in an external knowledge base and let the model read the result. The evidence in this area shows that retrieval alone does not close the gap. Even when the right entity or the right article is handed over, models often fail to pull the answer out of it or to chain several facts together, and they fall back on what they memorized during pre-training. We propose to train this reading-and-reasoning skill on large synthetic corpora built around fictitious visual entities. These are invented species, buildings and artworks, each with generated images and generated encyclopedia pages. Because no model has seen these entities before, the only way to answer a question about one is to find the matching document, read it, and reason step by step over what it says.

The opportunity is that synthetic data offers something real data cannot here: a guarantee that memorized knowledge is useless. That guarantee makes it possible to scale supervision for grounded, multi-step reading to millions of examples. The number of distinct fictional worlds and question structures becomes a variable we can control, so we can measure how it changes the model's tendency to follow evidence rather than its priors. The claim is testable on existing real-world benchmarks (INFOSEEK, Encyclopedic-VQA, MMhops, MC-Search). It is also falsifiable by one ablation: the same pipeline built on real entities.

Core Claim

Training on retrieval-augmented, chain-of-thought trajectories about fictitious multimodal entities teaches a vision-language model to rely on retrieved evidence over its parametric priors. Once fictitiousness rules out memorization, this evidence-reliance skill scales with the diversity of synthetic worlds and reasoning structures. Chain-of-thought here means the model writes out intermediate reasoning steps before answering. The skill should transfer to real knowledge-based visual question answering. Our specific prediction is that fictitious-entity training beats an equal-size synthetic corpus built from real Wikipedia entities, and that the gap is largest on long-tail and counterfactual-evidence questions, where priors are most misleading.

Four strands of prior work motivate this. Two benchmark papers isolate a reading failure that is separate from perception and retrieval. INFOSEEK reports that oracle entity recognition lifts accuracy only to 45.6% on its human split. Encyclopedic VQA finds that models struggle to use retrieved text even when it is correct. Work on knowledge conflict in multimodal long-chain reasoning probes hidden states during reasoning and shows that conflicts between vision, priors and text are detectable inside the model. It also finds that steering toward grounded evidence is easier than enforcing textual counterfactuals, so prior-override is a real, measurable failure. The text-only method PREREQ-TUNE supplies the key technique: fabricated biographies force a model to depend on the knowledge it is given rather than on pre-training. That work stays text-only and discards the knowledge at test time. We carry fictitiousness into the multimodal, open-book setting, where the knowledge is retrieved and read at inference.

Research Program

The data engine builds fictional worlds. Each world is a small knowledge graph of invented entities with typed attributes and relations, such as a fictional architect, the bridges she designed, their construction years, and the towns they sit in. A text-to-image generator produces several visually distinct views per entity. One view goes into the entity's encyclopedia page as its reference image, and the others are held out as query images. An LLM turns each graph neighbourhood into a Wikipedia-style article, and a verifier checks that every graph fact appears in the article text.

Questions are then sampled over graph paths. They come in the hop structures that MC-Search and MMhops-R1 define on real data: single-hop lookup, bridging chains that start from an image, and comparisons across entities. We borrow MC-Search's idea of checking that every hop is necessary. For comparison and arithmetic hops, such as "how many years apart were these built", the reasoning trace includes executable code. This follows Synthesize Step-by-Step, which turned chart reasoning into Python rationales but stayed in a closed world. Comparison questions are where MMhops-R1 is weakest, so this is where the code steps should matter most.

Each training trajectory has four parts. The model first matches the query image against several candidate entity images and pages, including generated near-duplicates as distractors. This deliberately drops MC-Search's top-1 retrieval assumption. The model then quotes the supporting sentence from the chosen page, chains to the next hop, executes any code, and answers. The retrieval context mixes the correct pages with distractors at graded difficulty, borrowing the noise levels of Wiki-R1.

The model is Qwen2.5-VL-7B. We train it with supervised fine-tuning on these trajectories, then optionally with reinforcement learning on the final answer.

The scaling study varies two quantities independently: the total number of trajectories, from 10k to about 2M, and the number of distinct fictional worlds and entity types. The research file's synthetic-data evidence, such as DiverGen, predicts that diversity, not volume, drives gains. One known risk is that aligned generators collapse toward typical outputs, as Verbalized Sampling documents. We therefore measure world diversity directly, through attribute entropy and image-embedding spread, and report scaling against it.

Baselines

The naive reference point is the untuned Qwen2.5-VL-7B given the same retrieved documents and prompted to reason step by step. This shows how much any training adds over prompting alone. A training-free prompting baseline is VCTP from Visual Chain-of-Thought Prompting, which loops over detect, caption and verify steps. It reports 46.4% validation accuracy on A-OKVQA with an OPT-66B backend. We run it with the same retriever.

The strongest existing competitor on real single-hop knowledge VQA is Wiki-R1, which trains with reinforcement learning on real Wikipedia under a retrieval-noise curriculum. It reports 44.1% overall accuracy on INFOSEEK and 47.8% on the Unseen-Question split, and 37.1% on Encyclopedic-VQA. It is the closest competitor because it also trains reasoning over noisy evidence, only with real entities. EntRAG reports 44.5 on INFOSEEK and 46.1 on Encyclopedic-VQA with a stronger entity retriever. That 46.1 likely reflects better retrieval, so we compare to it only when we use its retrieval output.

For multi-hop questions, the competitors are MMhops-R1, which reports 51.35% on Bridging and 22.01% on Comparison tasks, and SEARCH-ALIGN, which reports a +13.7 F1 gain on Qwen2.5-VL-7B on MC-Search. No absolute MC-Search score for SEARCH-ALIGN was found on Lacuna, so we will reproduce it.

To make the comparison fair, every trained baseline uses the same backbone, the same frozen retriever and knowledge base at test time, the same number of training examples and GPU-hours, and the same hyperparameter search budget. Beating Wiki-R1 means exceeding 44.1% on INFOSEEK under its retrieval setting. Beating MMhops-R1 means clearing 22.01% on Comparison tasks.

The ablation that isolates the key ingredient is a real-entity twin corpus. It uses the same generator, the same hop structures, the same distractor schedule and the same size, but builds questions from real Wikipedia entities and photos. The claim survives only if the fictitious corpus wins. A second ablation removes the images, replacing them with captions, to test whether visual matching against fictional candidates contributes anything beyond text-level grounding.

Why not just…?

Why not just synthesize questions from real Wikipedia, which is cheaper and avoids image generation? This is the main threat, and it is the twin-corpus ablation above. Real entities let the model answer correctly from memory while ignoring the retrieved page, so supervision rewards the wrong behaviour. PREREQ-TUNE's text-only results suggest this matters. If the twin corpus matches ours on accuracy and on the evidence-following measure described below, the fictitious ingredient is unnecessary, and we would report the work as a scaling study of real-entity synthesis instead.

Why not just improve entity linking and retrieval, since INFOSEEK names linking as the bottleneck? Better retrieval is complementary, not a substitute. With oracle linking, INFOSEEK accuracy is still only 45.6%. Encyclopedic VQA reports PaLM, a text-only model, at 82.3% with correct retrieval, while vision-language models with retrieval sit far lower. We evaluate under fixed oracle, EntRAG and noisy retrieval, so that reading gains are separated from retrieval gains.

Why not just run Wiki-R1-style reinforcement learning with a noise curriculum on more real data? Wiki-R1 improves INFOSEEK by 4 points over its baseline, but real Wikipedia limits how much data exists and still permits memorized shortcuts. We will add our corpus as a pre-stage to Wiki-R1 training. If the combination gives no gain over Wiki-R1 alone at equal compute, the synthetic stage adds nothing.

Why not just do the fictitious training in text and rely on the frozen vision encoder? The images-to-captions ablation tests this directly. We expect text-only training to help single-hop reading but not candidate disambiguation from query images, so the difference should show up on image-initiated chains in MC-Search.

Proposed Experiments

We evaluate on INFOSEEK (full and Unseen-Question splits), Encyclopedic-VQA (single-hop and two-hop), MMhops (Bridging and Comparison) and MC-Search (F1 and Hit per Step). The metric is accuracy under each benchmark's own protocol.

The decisive measurement is an evidence-following rate on counterfactual versions of these benchmarks. We edit the retrieved page for a real entity to state a different fact, such as a different construction year, and record how often the model's answer tracks the edit rather than the true fact. Alongside it we report a prior-override rate: how often the model answers with its prior despite the edit. The probes from the knowledge-conflict paper serve as an internal check that conflict signals inside the model decrease after training.

The central prediction is that fictitious training raises evidence-following well above both the real-entity twin and Wiki-R1, while matching or beating them on standard accuracy. We also expect the scaling curves to rise with world diversity at fixed volume and to flatten with volume at fixed diversity.

Two results would falsify the claim. The first is a twin corpus that matches ours on evidence-following. The second is a fictitious model that follows evidence but loses accuracy on real benchmarks, which would mean it learned to distrust all parametric knowledge, including correct knowledge. To catch the second failure, we also report accuracy on OK-VQA-style questions that need no retrieval.