Many questions about images need several steps of lookup. To answer "When did the architect of this building die?", a system first has to work out which building is in the photo, then find its architect, then find that person's death date. Current systems handle each step in words. A vision-language model writes the building's name, and a text search engine takes it from there. This works when the model can name what it sees. It fails, and fails for good, when the first-step entity is rare. A vision-language model (a network pre-trained to align images with text, such as CLIP or a multimodal LLM) can often see that the building in this photo looks like other photos of one particular building, yet be unable to produce that building's name. Every text-based chain then starts from a wrong or empty name.
We propose to stop forcing the first hop through words. The intermediate entity would travel as a vector instead: a question-conditioned hidden state read from a middle layer of a frozen vision-language model, mapped by a small trained projector into the embedding space of a multimodal entity index (Wikipedia-style pages with images). The system can then retrieve the page for an entity it cannot name. Once that page is in hand, later hops run in ordinary text, where text retrieval already works well. Our central bet is that frozen foundation models perceive more than they can name. If so, the "knowledge ceiling" that appears in multi-hop visual QA is partly a naming ceiling created by text-mediated pipelines, and a latent hop can get under it.
Core Claim
On multi-hop visual questions whose bridge entity the backbone cannot name, a latent visual hop should retrieve the correct bridge document far more often than name-then-search pipelines. On questions where the entity can be named, it should roughly match them, and it should run at close to single-retrieval latency. A second, falsifiable prediction is that the best layer to read from lies in the middle of the network, not at the top.
That prediction rests on two findings that come from different worlds. Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation shows that final-layer CLIP features suffer "semantic contamination": dominant objects bleed across image patches, and spatial coherence peaks in intermediate layers, around layer 9 of ViT-L. Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) finds, in text-only LLMs, that middle layers carry the "internal reasoning" stage. It maps those hidden states with an MLP into a dense retriever's space to fetch the second-hop document, reaching 0.352 accuracy on MuSiQue at about 2.8 s per query, against more than 20 s for an iterative generator. We take L-RAG's mechanism and move it to the place where it matters most: hops whose source is visual and whose target the model cannot verbalize. L-RAG never faces that case, because its bridge entities are always expressible in its own vocabulary.
Why the gap is real
The multimodal multi-hop systems in the file all mediate hops through text. OmniSearch plans sub-questions in language and solves them with a smaller multimodal model. MMhops-R1 trains Qwen2.5-VL-7B with reinforcement learning to alternate written thoughts with image and text search calls. MMAPG builds a planning graph whose nodes are verbalized thoughts and facts from frozen Llama-3 and LLaVA. QA-Dragon routes by domain and uses detectors.
Their reported failures point to the same place. QA-Dragon struggles to tell apart fine-grained, biologically similar classes such as plant species. MMAPG's own analysis notes that a misread visual entity poisons every downstream branch. WebQA showed that turning images into text is a lossy bottleneck, with models at about 50% on text-based queries but about 28% on image-based ones. These papers motivate the proposal.
Two works from the foundation-model side supply the method's justification. Visual features can be matched to concepts without names: MASK builds visual prototypes for words, and W2VV regresses text directly into visual feature space. ConText-CIR constrains the design. Long text inputs make CLIP fail to ground the specific noun phrase. Our query vector must therefore be conditioned on the part of the question that names the bridge, such as "the architect of this building", and not on the whole question.
Research Program
The system has four parts. The backbone is a frozen Qwen2.5-VL-7B, chosen so the comparison with MMhops-R1 is like-for-like. The index is a multimodal entity index: entity pages, each embedded with a frozen CLIP-family image–text encoder from the page's images and lead text. The projector is a two-layer MLP. It takes the backbone's hidden state at a chosen layer, pooled over the image tokens with the question-to-image attention for the bridge phrase, and outputs a vector in the index space.
The projector is trained with a contrastive loss to retrieve the gold entity page. Training data comes from INFOSEEK (single-hop visual entity grounding) and from the first-hop annotations in the MMhops training split. Only the projector is trained. After the bridge page is retrieved, its text goes into the context, and the remaining hops run as standard text retrieval with the same backbone, in the style of IRCoT.
A gate decides which route the first hop takes. If the backbone names the entity with high confidence, the hop uses a text query. Otherwise it uses the latent query. We keep the gate simple, a threshold on the entropy of the generated name, borrowing the uncertainty idea of DRAGIN without its attention machinery. The goal is that the system costs nothing on easy questions.
The analysis that gives the work its meaning is a stratification. We prompt the frozen backbone to name each bridge entity and check the answer against the gold entity. This splits every evaluation set into a nameable slice and an unnameable slice, with a second split by entity frequency as a check on the first. On the unnameable slice we also run a linear probe. It asks whether the hidden state still identifies the entity within a candidate set. This tests directly whether perception survives where naming fails. If the probe is at chance, the knowledge ceiling described by Predicting Visual Features from Text for Image and Video Caption Retrieval is genuine at that point, and no latent trick will help there.
Baselines
The naive reference point is the backbone answering with no retrieval. For scale, the file reports GPT-4o at 36.62% accuracy on MMhops bridging tasks and 8.76% on comparison tasks (MMhops-R1), and base GPT-4V at 30.25% F1-Recall on Dyn-VQA (OmniSearch). We run our own no-retrieval Qwen2.5-VL-7B under the same prompts.
The strongest existing method on our main benchmark is MMhops-R1. It reports 51.35% accuracy on MMhops bridging tasks, 22.01% on comparison tasks, and 33.2% on INFOSEEK. It is the right competitor because it shares our backbone and already has both image and text search tools. Any gain therefore has to come from how the first hop is expressed, not from model scale. We reproduce it with its released recipe and give it our entity index as one of its tools, so both systems search the same corpus. We cap both at the same number of retrieval calls, and we report wall-clock latency.
On Dyn-VQA the comparison is OmniSearch at 50.03% F1-Recall. Dyn-VQA requires live web search, so there we swap only OmniSearch's sub-question solver for our latent hop and keep its planner. On WebQA the reference is MMAPG at 65.9 QA-accuracy, run on the image-query subset.
Beating these baselines means two things. First, a statistically clear gain in first-hop recall@5 and in final accuracy on the unnameable slice. Second, no loss larger than about one point on the nameable slice. Overall accuracy is reported as well, but it is not the decisive number, because the nameable slice may dominate it.
Three ablations isolate the key ingredient. The first reads the final layer instead of a middle layer; it tests the semantic-contamination prediction. The second removes question conditioning, pooling over all image tokens; this checks whether ConText-CIR's interference concern applies here. The third is the most important. It replaces the hidden state with a CLIP embedding of an image crop chosen by the backbone's attention. This tells us whether the backbone's internal state adds anything beyond "crop the right region and do image search."
Why not just…?
Why not just let the model name the entity and run text multi-hop retrieval? This is what IRCoT-style and MMhops-R1 pipelines do, and on the nameable slice it should be as good as anything. Its weakness is structural. When the name is wrong, the text query retrieves confidently wrong pages, and the chain diverges. That is the cascading failure described for CoRAG and MMAPG. The stratified evaluation measures exactly how much accuracy this route loses on unnameable entities. If that slice turns out to be tiny on MMhops and Dyn-VQA, the proposal's practical value shrinks to a niche, and we will say so.
Why not just do reverse image search on the raw image or a crop? This is the real threat. OmniSearch already has an image-to-image tool. For single-object photos, raw-image search may find the entity without any learned projector. Our argument is that real questions pick out one entity among several. Whole-image search then retrieves the visually dominant object, which is the retrieval-level version of the patch bleeding seen in CLIP's top layers. The crop ablation settles this. If an attention-chosen crop plus CLIP image search matches the latent hop, we report that region conditioning, not hidden states, is the contribution, and we simplify the method accordingly.
Why not just fine-tune or scale the backbone? Larger models do not close the gap in the file. GPT-4o trails a 7B retrieval-trained model on MMhops. The post-hoc and training interventions surveyed alongside CLIPSeg clean up representations but cannot add entities absent from pre-training. Fine-tuning the backbone on INFOSEEK names would teach it only the training entities. The projector instead learns a general mapping into an index that can grow without retraining. We test this with entities held out from projector training entirely.
Decisive Checks
The work stands or falls on four results. The first is first-hop recall@5 on the held-out-entity, unnameable slice of MMhops bridging and INFOSEEK. Here the latent hop must beat name-then-search clearly and must not be matched by the crop-plus-image-search ablation. The second is the layer sweep, which should peak in the middle of the backbone. A peak at the final layer would weaken the transfer of the CLIPSeg and L-RAG findings. The third is the probe, which should show that entity identity remains decodable on items the model cannot name. Without that, the hypothesis is false. The fourth is end-to-end accuracy on MMhops comparison tasks, where MMhops-R1 sits at 22.01%. Comparison questions need several visual bridges, so they should gain the most if visual hops are where chains break.
As a cost check, the latency of the gated system should stay near L-RAG's single-pass regime rather than the more than 20 s cost of fully iterative generation.
References
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
- MMhops-R1: Multimodal Multi-hop Reasoning
- MMAPG: A Training-Free Framework for Multimodal Multi-hop Question Answering via Adaptive Planning Graphs
- QA-Dragon: Query-Aware Dynamic RAG System for Knowledge-Intensive Visual Question Answering
- WebQA: Multihop and Multimodal QA
- Multimodal Aligned Semantic Knowledge for Unpaired Image-text Matching
- Predicting Visual Features From Text for Image and Video Caption Retrieval
- ConText-CIR: Learning from Concepts in Text for Composed Image Retrieval
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Chain-of-Retrieval Augmented Generation