One Pass, Two Embeddings: Training a Cheap Multimodal Next-Hop Retriever and Testing It Where Memorization Cannot Help

Checking a claim like "the team whose logo is on this player's shirt was founded in the city where the 1936 Olympics were held" takes several steps. A system has to read a detail in the image, turn it into a search for the next piece of evidence, and continue from there. Today's best multimodal fact-checkers handle these chains by orchestrating large commercial models with web search, reverse image search and debate rounds. These pipelines are slow, expensive and fragile. The cheaper alternative is to train a dense retriever, which turns claims and evidence into vectors and finds matches by similarity. Training one by contrastive learning, which pulls matching pairs together and pushes mismatches apart, usually needs two forward passes or two separate encoders per example. That cost is hard to bear on a multi-billion-parameter multimodal model. A further problem is that much of the apparent skill on existing multi-hop benchmarks may come from memorized text rather than from looking at the image.

We propose to train a multimodal language model so that a single forward pass over any evidence item produces two vectors. The first says what this item is. The second says what the next piece of evidence should look like. Linking the second vector of hop $k$ to the first vector of hop $k+1$ gives a contrastive objective that costs one pass per item and needs no second encoder. We then evaluate the retriever on two kinds of data: claims whose bridge entity is visible only in the image, and claims the model is unlikely to have memorized. A text-only shortcut cannot succeed on either kind.

Core Claim

The mechanism adapts a trick from CSE-SFP. That paper uses the causal attention mask of a decoder model to obtain two different sentence embeddings in one pass. It formats the input as prefix, sentence, a first readout token, a suffix instruction, and a second readout token. Because the first readout cannot see the suffix, the two readouts act like two views of the sentence. On Llama-3 8B with QLoRA this cut contrastive training from 280 to 159 minutes and saved about 8GB of GPU memory compared with two-pass SimCSE. CSE-SFP is text-only and trains generic sentence similarity.

We give the two readouts different jobs. The input sequence is image or table tokens, accompanying text, the claim context, readout $r$, a suffix such as "the next fact needed to verify this is about", and readout $q$. Readout $r$ sees only the evidence and becomes its index vector. Readout $q$ sees the evidence plus the instruction and becomes a query for the next hop.

Training uses the gold evidence chains in MMCV, the multi-hop multimodal claim benchmark from Piecing It All Together. For a batch of $N$ evidence items drawn from many chains, the loss is

$$ \mathcal{L} = -\sum_{k} \log \frac{\exp(\mathrm{sim}(q_k, r_{k+1})/\tau)}{\sum_{j=1}^{N} \exp(\mathrm{sim}(q_k, r_j)/\tau)}, $$

where $q_k$ is the next-hop readout of item $k$, $r_{k+1}$ is the index readout of the true next item in the same chain, $r_j$ ranges over every item in the batch, $\mathrm{sim}$ is cosine similarity, and $\tau$ is a temperature. Every item appears once in the batch and supplies both a query and a key. The score matrix is therefore $N \times N$ at the cost of $N$ forward passes, where a separate query encoder and document encoder would need $2N$.

This connects to Optimizing Multi-Hop Document Retrieval Through Intermediate Representations, which showed that a text model's middle-layer hidden states already encode the next hop. Its method, L-RAG, retrieves from those states through a small projection head and reaches about 2.8 seconds per query, against more than 20 seconds for iterative generation. We read our readouts from those middle layers and test whether the same property holds when the bridge entity is visual, a case L-RAG never covers.

The claim we will test is narrow. On chains where the next-hop entity can only be read from the image, a retriever trained this way should beat a pipeline that first captions the image and then runs text retrieval. The advantage should be concentrated on claims the model has not memorized.

Research Program

The work has three parts.

The first part builds the retriever on an open multimodal model of the LLaVA family, the family used in MMCV and in How to Train Your Fact Verifier. Training uses QLoRA adapters, following CSE-SFP. We will choose the readout layer by sweeping middle layers on a validation split, guided by L-RAG's finding about where next-hop information sits. At test time the retriever runs hop by hop: the query vector of the current item retrieves from a mixed index of Wikipedia passages, images and tables, and the retrieved item's own query vector drives the next hop. A final verification call to the same model produces the verdict, so only one generation step happens per claim.

The second part is a contamination-controlled evaluation, which addresses the open puzzle in MMCV. That paper attributes the 27% human-model gap on 4-hop claims to failures of visual perception. It also reports that LLaVA-v1.5-7B scored higher closed-book than open-book, which suggests the model was answering from memory. Both Text and Images Leaked! shows that multimodal benchmark leakage often enters through the language backbone's text-only pre-training. MMCV is built from Wikipedia and MultimodalQA, so it is exposed to exactly that route.

We will therefore score every MMCV test claim for likely memorization with two signals. The first is closed-book accuracy across several backbones. The second is Min-K% Prob membership scoring, which Unveiling Causal Reasoning in Large Language Models used to show that its post-cutoff probe had lower pre-training likelihood. We split claims into high-memorization and low-memorization strata. We also annotate a "visual-bridge" subset, where the hop-2 entity appears in the image but not in any caption or alt text.

The third part adds fresh data. We will evaluate on the multimodal portion of ClaimReview2024+ from DEFAME, which contains 300 claims dated after GPT-4o's training cutoff. We will also build a probe set of a few hundred new chains from news images published after the backbone's cutoff, with templated hop structure in the spirit of DyVal. This probe set is a small test of controllable depth, not a general benchmark. It lets us report accuracy as a function of hop count on data the model cannot have seen.

Baselines

Closed-book verifier. The naive reference is the same backbone with no retrieval, prompted to verify the claim directly. Vanilla GPT-4o on ClaimReview2024+ gives the scale of the problem: DEFAME reports 35.2% accuracy for it. On MMCV's high-memorization stratum we expect closed-book performance to be deceptively strong. That expectation is the reason for the stratification.

Strongest open-book result on MMCV. No reported result on Lacuna was found for a trained multimodal multi-hop retriever on MMCV. The strongest number in the file is Gemini 1.5 Flash in the open-book setting, at 70.92% average F1 (Piecing It All Together), with human experts at 80–90%. We will re-run this setting with our backbone on the same retrieval corpus, so that every difference comes from the retriever. Success means higher F1 than that matched re-run on the 3- and 4-hop claims in the low-memorization stratum, at a fraction of its latency.

Caption-then-text retrieval. This is the competitor that matters most. The same backbone writes a caption and reads text from each image, and the resulting text goes into IRCoT (Interleaving Retrieval with Chain-of-Thought Reasoning) and into a text-only L-RAG trained on the same chains. IRCoT reports recall gains of 11–21 points and QA gains of up to 15 F1 on text benchmarks. L-RAG reports 0.352 accuracy on MuSiQue. We will reproduce L-RAG's MuSiQue number as a sanity check before trusting the multimodal comparison. Training data, adapter size and hyperparameter search budget will match our method. Beating this baseline means higher hop-2 retrieval recall and final F1 on the visual-bridge subset. Matching it elsewhere is acceptable.

Agentic pipeline as a reference. DEFAME reaches 69.7% on ClaimReview2024+. MAD-Sherlock (MAD-Sherlock) reports 85.2% on VERITE. These systems use commercial models and live web tools, so the comparison is not resource-matched. We report them as an accuracy ceiling against which we plot our cost and latency. The bar is to reach a substantial fraction of DEFAME's ClaimReview2024+ accuracy at a small fraction of its per-claim time.

Ablations. Three ablations isolate the key ingredient. The first is a two-pass dual-encoder version with separate query and document passes, trained on identical data to the same number of steps. It measures the overhead saved, with CSE-SFP's 43% speedup as the reference point, and checks whether accuracy changes. The second removes the suffix, so a single readout serves as both query and key. This tests whether the "what next" instruction does any work. The third uses an off-the-shelf CLIP-style image-text encoder as the retriever with no training.

Why not just…?

Why not just caption the images and run text multi-hop retrieval? This is the real threat. For many MMCV chains, a good caption probably names the bridge entity, and text retrievers are mature. The evidence against captioning is indirect. MMCV attributes cascading failures to missed fine-grained cues such as logos, uniforms and table headers. VERITE is constructed so that neither modality alone determines the label. Neither result directly compares hidden-state retrieval with caption retrieval. Our visual-bridge subset is designed to settle the question.

If caption-then-L-RAG matches our retriever there, we will report it plainly. The multimodal contribution would then shrink to a training-cost result. We would conclude that the bottleneck lies in perception, which the text pipeline shares, rather than in how perceived content is represented.

Why not just prompt a strong commercial model with search tools? DEFAME shows that this works: 69.7% on fresh claims. DEFAME's own discussion, and the MultiReflect and MAD-Sherlock papers, also describe the cost: many sequential calls, scraping that breaks when sites change, and latency of the kind L-RAG measured at more than 20 seconds for iterative generation. We do not claim to beat these systems on accuracy. We claim a different point on the accuracy-cost curve, and we report both axes.

Why not just train a standard dual encoder with two passes? It would likely work, and that is why it is an ablation and not a strawman. The question is whether one pass loses accuracy. CSE-SFP found it did not for sentences. Patch-level contrast in All Patches Matter raised CLIP-Large memory from 16GB to 19GB, which shows how quickly dual-view costs grow with image tokens. If the two-pass version is clearly more accurate, our method's value becomes purely a matter of cost, and we will state it that way.

Why not just evaluate on fresh data and skip the memorization analysis? Fresh multimodal multi-hop data is scarce: ClaimReview2024+ has only 300 claims, and our probe set will be small. Stratifying MMCV lets us use its scale while still separating evidence use from recall. Training and testing on data drawn from Wikipedia could make every method look good for the wrong reason.

Decisive Experiments

The first check is hop-2 retrieval recall and final F1 on MMCV's visual-bridge subset, comparing our retriever with caption-then-L-RAG and caption-then-IRCoT under matched budgets. This test decides whether the idea survives.

The second check is the interaction between method and memorization stratum. The prediction is that gains over the closed-book and caption baselines are small on high-memorization claims and large on low-memorization claims, the fresh probe set, and multimodal ClaimReview2024+. If gains are uniform, the method's improvement is unrelated to memorization, and the memorization explanation for the multi-hop gap gains no support from this work.

The third check is an image-swap test. We replace the image with a mismatched one and count how often the retrieved hop-2 item and the verdict change. A model that relies on the image should change its answer. A model answering from memorized text will not.

The fourth check measures cost: training time and peak memory against the two-pass ablation at equal batch size, and per-claim latency against the agentic references. We report accuracy against hop count from 1 to 4 for all methods, so that the 4-hop gap reported in MMCV can be tracked directly.