Many question-answering systems that must reason over both images and text turn each image into a short caption, then hand that caption to a language model that searches a document collection. This works poorly when the image is the bridge to the next step. Take the question "where did the scientist who first described the bird in this photo work?" The caption has to name or describe the bird precisely enough that a search engine returns the right species page and not one of a dozen look-alikes. Captioners are trained to imitate average human descriptions, so they write "a small brown bird on a branch." That caption is true, but it cannot tell the target apart from its neighbours, and every later reasoning step inherits the wrong document.
We propose to train and decode captions against the search engine they will actually be fed to. The caption is rewarded when the real text retriever ranks the correct next-hop document above the documents for visually confusable entities, and those confusable entities are mined from the retrieval collection itself rather than taken from a fixed list of distractors. We test this on multi-hop chains that are generated fresh for each evaluation. Each chain attaches invented facts to fine-grained images, so a model cannot reach the answer from memorized world knowledge, and the benefit of distinctive captions cannot be hidden by data contamination.
Core Claim
The "lossy bottleneck" that WebQA reported is the multi-hop form of a known captioning failure: training by maximum likelihood pushes captions toward safe, generic wording. WebQA converted images into standard captions or tags, fed them to GPT-3, and found that image-based queries reached only about 28% accuracy against more than 94% for humans. Likelihood training rewards the average description, a point surveyed by Elliott et al.. Reinforcement learning on CIDEr creates a vocabulary bottleneck that penalizes rare, distinctive words, as shown by Honda et al.. We claim three things. First, when the caption is the query for the next hop, what matters is how distinctive the caption is relative to that retriever and collection, not how long or detailed it is. Second, a captioner optimized for this target will beat length-matched detailed captions and fixed-distractor pragmatic captioners on next-hop recall and on final answer accuracy. Third, the advantage will be largest exactly where memorized knowledge cannot rescue a vague caption. The third claim is also why contamination belongs in the study: on static benchmarks about well-known entities, a strong reader model can guess the bridge entity from memory, which makes generic captions look better than they are.
Relation to Prior Work
The distinctive-captioning literature supplies the method. Luo et al. add a reinforcement-learning reward for captions that a pre-trained image–text retrieval model matches to the target image rather than to other images; this raised accuracy on hard MS COCO pairs from about 71% to about 80%. DifDisCap contrasts each target against K hard negative images retrieved with VSE++. Vedantam et al. do the contrast at decoding time, trading target likelihood against distractor likelihood. PICL steers an OFA captioner token by token with a frozen CLIP listener, meaning a model that scores how well each candidate image matches the text. PICL reaches 65.7% human retrieval accuracy on ImageCoDe, against 54.0% for Emitter-Suppressor and 50.7% for incremental rational-speech-act decoding.
All of these methods have two features in common. The listener is an image–text matching model, and the distractors are a small fixed set of other images. We change both. Our listener is the downstream text retriever operating over documents. Our distractors are the documents for confusable entities, retrieved from the collection at training time, and at test time they come from visually nearest neighbours in the image pool. This targets the open problem that distinctive captioning has never scaled beyond small, predefined distractor pools. It also asks a different question. We do not ask whether a human can pick the image out of ten candidates. We ask whether the caption brings back the right document for the next reasoning step.
The multi-hop retrieval literature supplies the pipeline and motivates the stakes. IRCoT uses each chain-of-thought sentence as the next search query. CoRAG trains a model to generate sub-queries and intermediate answers, and it reports that one wrong intermediate retrieval sends the whole chain off course. Both systems are text-only. We keep IRCoT as the reasoning loop and change only how the image is turned into a query.
The contamination literature shapes the evaluation. DyVal generates reasoning problems on the fly from directed acyclic graphs, but only for symbolic tasks, with no retrieval and no images. MuSiQue showed that static multi-hop sets allow shortcuts: a single-paragraph model reaches 64.8 F1 on HotpotQA but only 32.0 on MuSiQue. MuSiQue removes shortcuts with a connectivity filter, which we reuse. We carry DyVal's fresh-generation idea into visual multi-hop retrieval, so that shortcuts from memorized facts and shortcuts from skipping a hop are ruled out by construction.
Research Program
The captioner is a pretrained vision-language speaker, OFA base as in PICL, conditioned on both the image and the question. Conditioning on the question lets the caption concentrate on attributes that bear on the relation being asked about. We fine-tune it with self-critical REINFORCE, following Luo et al., using the reward
$$R(c) = \log \frac{\exp s(c, d^{*})}{\sum_{d \in \{d^{*}\} \cup \mathcal{N}} \exp s(c, d)} + \beta \log p_{\text{LM}}(c).$$
Here $c$ is the caption and $s$ is the frozen text retriever's relevance score. $d^{*}$ is the gold next-hop document, and $\mathcal{N}$ is the set of documents for the entities most visually confusable with the target. We mine $\mathcal{N}$ from the collection by retrieving with the target's image embedding and keeping hits whose linked entities differ from the gold one. The last term uses a frozen language model's likelihood $p_{\text{LM}}(c)$, weighted by $\beta$, to penalize the ungrammatical "bag of words" drift that both PICL and Vedantam et al. report when discriminativeness is pushed hard.
At test time the gold document is unknown. We therefore also run a decoding-only variant in the style of PICL. Its listener rewards captions that a CLIP listener matches to the target image rather than to its nearest neighbours in the image pool. This variant separates what training against the actual retriever adds beyond pragmatic decoding.
The evaluation generator produces "fresh chains." We start from confusable image groups: CUB-200-2011 species used in Vedantam et al., and static-image sets from ImageCoDe. For each evaluation seed, we sample a small graph of invented facts, attach one document to each entity, and write two- to four-hop questions in which the image identifies the first entity. We apply MuSiQue's connectivity filter to discard any question that an adversarial model answers without the visual hop. The same chain is also rendered with real encyclopedia facts in place of invented ones. The difference in accuracy between the real and invented versions of the same chain directly estimates how much a system leans on memorized knowledge.
Baselines
The naive reference point is a standard likelihood-trained OFA caption, not conditioned on the question, passed into IRCoT with the same retriever and reader. This reproduces the lossy pipeline that WebQA diagnosed. The main result is that our captioner beats this reference on next-hop recall@5 and final answer F1 on fresh chains.
The most dangerous competitor on the lossy-bottleneck question is the argument that detail, not discriminativeness, is what was missing. We test it with long, detailed captions from the classifier-free guidance captioner of Kornblith et al., which is shown to make captions more descriptive even as n-gram scores drop. We run it at matched token budgets of 20, 40 and 80 tokens and also unconstrained. To count as beating it, our method must win at every matched budget, not only at the budget where we are strongest.
The strongest existing discriminative method is PICL with its fixed-distractor CLIP listener. It is the only baseline with a reported number on this kind of problem: 65.7% human retrieval accuracy on ImageCoDe. We run PICL with the same OFA speaker, the same decoding compute (number of candidate tokens rescored), and a matched sweep over its informativity weight.
We add two end-to-end references that skip text captions entirely. The first is a vision-language model that reads the image and the retrieved documents directly inside the IRCoT loop. The second is CLIP image-to-document retrieval for the first hop. Both receive the same retrieval budget and the same number of reasoning steps. On WebQA the best fine-tuned vision-language models reach about 24% on combined accuracy and fluency, and image-based queries reach about 28% (WebQA). The file reports no number for either system on fresh chains, and for text reasoning the strongest pipeline figure is CoRAG's increase in MuSiQue retrieval recall from 29.0% to 47.1%.
Three ablations isolate the key ingredient. We replace mined confusable distractors with random ones, we remove question conditioning, and we swap the retriever-based reward for a CLIP-listener reward. If the CLIP-listener version matches the retriever-based one, the claim about the retriever as listener fails.
Why not just…?
Why not just write longer, more detailed captions? This is the most serious threat, because WebQA's diagnosis fits "too little detail" as well as it fits "too generic." Long captions, however, spend most of their tokens on attributes the look-alike entities share, such as brown feathers and a branch. That dilutes the query for dense retrievers and inflates the growing context that CoRAG already identifies as a cost. The length-matched comparison settles the question. If detailed captions match ours at equal length, we will report that detail is sufficient and drop the discriminative claim.
Why not just use an end-to-end vision-language model and skip captions? End-to-end models avoid the bottleneck when reading. They still need a text query to search a document collection, and they generate that query by the same likelihood decoding that produces generic wording. WebQA's end-to-end models were not better, at about 24%. If the end-to-end baseline nonetheless wins on fresh chains, that is a legitimate negative result for pipelines with a caption bottleneck, and we will report it.
Why not just let the reasoning loop rewrite a bad query after seeing what it retrieved? IRCoT-style iteration does allow repair. But a generic first query returns confident-looking wrong documents, and the error-cascade analysis in CoRAG shows that chains rarely recover from those. We give the IRCoT baseline extra rewrite steps equal to our compute budget. That tests directly whether iteration alone closes the gap.
Why not just deduplicate static benchmarks instead of generating fresh chains? Deduplication removes overlap in the test items, but not the model's knowledge of the entities they mention. A reader that recognizes a famous landmark from its name alone needs no distinctive caption. The comparison between real and invented facts on otherwise identical chains measures this effect directly, and static WebQA scores are reported alongside for continuity.
Decisive Checks
Three outcomes would confirm the proposal, and each could fail. First, on fresh chains, retrieval-contrastive captions must improve next-hop recall@5 and final F1 over both PICL and length-matched detailed captions. Second, the gap between invented and real facts must be larger for generic captions than for ours. That result would show that static benchmarks overstate generic captioners because the reader supplies memorized knowledge. Third, our method's gain should grow as the number of confusable entities in the collection increases from 5 to 500 per group, while fixed-distractor PICL levels off.
Fluency is checked with frozen language-model perplexity and human ratings on the 5-point scale PICL used, where PICL scored 4.07. Human retrieval accuracy on ImageCoDe is also reported, so that any drop in single-image distinctiveness is visible. If the gains hold on WebQA's image-based subset as well as on fresh chains, the lossy bottleneck can be reinterpreted as a problem of caption objective rather than of modality.
References
- WebQA: Multihop and Multimodal QA
- Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
- Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning
- Discriminability Objective for Training Descriptive Captions
- Group-Based Distinctive Image Captioning with Memory Difference Encoding and Attention
- Context-Aware Captions from Context-Agnostic Supervision
- Pragmatic Inference with a CLIP Listener for Contrastive Captioning
- Image Retrieval from Contextual Descriptions
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Chain-of-Retrieval Augmented Generation
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Classifier-free guidance makes image captioning models more descriptive