Does the Image Fade Before the Search Fails? Attributing Multimodal Multi-Hop Errors

Picture an AI assistant that looks at a photo and then runs several web or database searches to answer a question about it. An example question: "When was the architect of the building in this picture born?" Each search returns text snippets, and those snippets pile up in the model's working memory. This proposal tests a specific suspicion. As the text piles up, the model stops paying attention to the photo. Its later searches then drift away from what the photo actually shows. What gets reported as a "retrieval failure" (the search never found the right document) may often be a perception failure one step earlier: the model forgot what it was looking for. When that happens, the model tends to invent a plausible connecting fact to keep its reasoning going. That invented fact is a hallucination, and it then contaminates every later step.

Two findings motivate the test. For text-only search agents, Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents shows that failure is dominated by never retrieving the key evidence. Agents that found gold evidence were right 73.8% of the time, versus 2.2% for agents that missed it. Failed runs were 2–3x longer, with snippets filling 66–85% of the context. Separately, MM-Snowball shows that in multi-turn image dialogue, accumulating text suppresses visual evidence. They call this "visual fading." It is reversible: accuracy recovers when the model is told to re-examine the image. Nobody has asked whether these two effects are the same failure seen from two sides. We propose a counterfactual replay method that splits each failed multimodal search into three causes: a retrieval gap, a utilization gap, and a new "anchor gap." We then use the split to decide which simple interventions are worth keeping.

Core Claim

We claim that a measurable share of failures on multimodal multi-hop benchmarks are anchor-gap failures. In these cases the agent's queries lose sensitivity to the input image as retrieved text accumulates. Restoring the image's influence at that step, and changing nothing else, would have led to retrieving the correct bridge document.

If this is true, it has three consequences. Methods that improve the retriever or train better search policies, such as the reinforcement-learned agent in MMhops-R1, are partly treating a symptom. Cheap fixes for visual attention should cut invented bridge facts in search settings, not only in closed dialogue. And in-context demonstrations, which usually help agents, carry a hidden cost in this setting. Their text adds to the pressure that makes the image fade.

The claim can be falsified. If replaying failed steps with the image restored almost never changes the retrieved evidence, the anchor gap is negligible. In that case the text-agent diagnosis carries over unchanged to multimodal agents, and that is itself worth reporting.

Research Program

Measuring image reliance. At every step of a trajectory we compute image sensitivity. This is the divergence between the agent's next-action distribution (its search query or its choice of tool) under the real image and under a replaced image, holding the text context fixed. We use two replacements, a blank image and a same-category distractor. As a correlational companion measure, we record the share of attention that goes to image tokens. The fading hypothesis predicts that image sensitivity falls as a function of how many snippet tokens have accumulated, not as a function of step count. Varying snippet length independently of the number of hops lets us separate those two explanations.

Attributing failures by replay. For each failed trajectory we find the first step at which the agent's query stops targeting the next gold entity. MMhops is organized as 3–4-hop bridging chains from a single image. We will use its per-hop structure if it is exposed; otherwise we annotate a sample of chains by hand. At that divergence step we run three replays, each changing one thing:

  • Anchor replay re-injects the image at the current position and leaves the text untouched.
  • Retrieval replay inserts the gold document into the search results.
  • Utilization check confirms whether the gold document was already present in context and was ignored.

Each failure is then assigned to one category. It is an anchor gap if anchor replay alone recovers the correct query and document. It is a retrieval gap if only gold insertion helps. It is a utilization gap if the evidence was already present. This extends the text-only split into retrieval and utilization gaps from Diagnosing Search Behavior, which has no visual channel.

Measuring invented bridges. Alongside the attribution, we log every intermediate sub-answer the agent states. We mark it as an invented bridge if neither the image nor any retrieved document supports it, judged by an entailment model plus human spot checks. We then test whether invented bridges cluster right after drops in image sensitivity.

Demonstrations as controlled text pressure. In-context learning is the model's ability to pick up a task from worked examples placed in its prompt. We use it here as a dose knob. Adding 0, 1, 2 or 4 full multi-hop demonstrations to the prompt raises the text load in a controlled way before the image enters. We test whether this speeds up fading even as it improves query formatting.

We then add one arm based on In-Context Learning Creates Task Vectors. That work shows demonstrations can be compressed into a single intermediate-layer activation and "patched" into a query-only forward pass, retaining 80–90% of full in-context accuracy. We patch a demonstration-derived vector instead of prepending demonstration text. The aim is to keep the demonstrations' guidance without their tokens. This arm carries real risk. Task vectors have so far been validated only on single-token outputs, and One Task Vector is not Enough suggests one vector may not capture a multi-step policy. We therefore apply the patch only at tool-choice and query-generation positions, and we treat failure of this arm as an informative result.

Backbone. All internal measurements use Qwen2.5-VL-7B, the backbone of MMhops-R1, so that comparisons with it are like-for-like.

Baselines

Naive reference. A prompted ReAct-style agent (one that alternates written reasoning with tool calls) on Qwen2.5-VL-7B. It has the same image_search, text_search and answer tools and the same local knowledge base as MMhops, and it receives no demonstrations and no re-anchoring. This fixes the base rate of each failure category.

Strongest trained competitor. MMhops-R1, which learns tool use with preference-optimization reinforcement learning. It reports 51.35% accuracy on MMhops Bridging tasks and 22.01% on Comparison tasks. The same paper reports GPT-4o at 36.62% and 8.76%, and Gemini-2.5-flash at 46.58% on Bridging (MMhops-R1). We will use the released checkpoint if one is available, and otherwise reproduce it with the paper's recipe. It gets the same tools, the same hop budget and the same evaluation split. Our attribution is also run on its trajectories, which tells us whether reinforcement learning shrank the anchor gap or only the retrieval gap. A training-free re-anchoring intervention on the prompted agent counts as beating it if it reaches or exceeds 51.35% on Bridging. Separately, if the anchor-gap share inside MMhops-R1's own failures is substantial and our intervention reduces it, that shows the trained policy leaves fading unaddressed.

Strongest planning competitor. On Dyn-VQA we compare against OmniSearch. OmniSearch is a planning agent with sub-question solvers and web, image-to-image and text-to-image retrieval. It reports 50.03% F1-Recall, against 30.25% for GPT-4V and a human level of 55% (Dyn-VQA / OmniSearch). Its authors describe "thinking traps," where the planner fixates on wrong visual cues. Our attribution tests directly whether those traps are anchor gaps.

Strongest mechanism-level competitor. CAVR from MM-Snowball. It re-injects visual embeddings when an intermediate-layer uncertainty signal fires, and blends the model's output scores with those of a history-free image-only pass. We port it unchanged into the agent loop with its original thresholds, and then with thresholds tuned on our development split under the same budget as our intervention. We found no reported result on Lacuna for CAVR in a retrieval-augmented or agentic setting.

Ablations that isolate the key ingredient. The key ingredient is re-anchoring triggered by measured image sensitivity. We compare it against three alternatives:

  • Periodic re-anchoring at every step, which tests whether the trigger matters at all.
  • Random re-anchoring at matched frequency, which tests whether timing matters.
  • Snippet compression with no re-anchoring, which tests whether reducing text pressure alone is enough.

For fairness, all arms share one backbone, one retriever, one hop cap, and an equal tuning budget of the same number of development-set configurations.

Why not just…?

Why not just re-insert the image at every step? This is the most serious threat. MM-Snowball's V-shaped recovery suggests that simply re-showing the image works. If periodic re-insertion matches sensitivity-triggered re-anchoring on accuracy and on invented-bridge rate, we will report that and recommend it. The core contribution would still stand, because it is the attribution: showing that the anchor gap exists and how large it is justifies re-insertion in search agents, where nobody currently does it. The trigger only earns its place if it wins on cost (fewer image tokens) at equal accuracy, or if repeated re-insertion measurably hurts use of the retrieved text. We measure both.

Why not just train a better search policy, as MMhops-R1 or FrugalRAG do? FrugalRAG learns when to stop searching and reaches 68.47% on HotpotQA with 2.05 searches, but it is text-only and remains bounded by its retriever. MMhops-R1 still drops to 22.01% on Comparison tasks. These are the multi-image settings where competition for visual attention should be worst. Reinforcement learning optimizes the final answer and does not report why trajectories fail. Running our attribution on MMhops-R1's own trajectories tells us whether training closed the anchor gap. If it did, that finding argues for training-based fixes, and we will say so.

Why not just shorten or summarize snippets? If fading is driven purely by text volume, compressing snippets could be enough, and it is cheaper than touching the image pathway. The snippet-compression ablation tests this directly. The prediction that separates the two explanations is this: compression should restore image sensitivity only if fading depends on token count, whereas anchor replay should help even at matched context length.

Why not just score grounding with embeddings and stop early? The cross-modal grounding score of Adaptive Termination in Agentic Multimodal RAG reports 90.0% faithfulness on PathVQA. It has two limits here. It measures topical similarity rather than logical entailment, and it was evaluated on only 30 queries. More importantly, stopping earlier does not repair a query that has already drifted from the image. We include its stopping rule as an extra arm on Dyn-VQA.

Proposed Experiments

The first decisive check is the attribution on MMhops Bridging and Comparison, run for the prompted agent and for MMhops-R1. We report the fraction of failures in each of the three categories with bootstrap confidence intervals. We expect the anchor-gap share to be larger on Comparison tasks, which involve multiple images.

The second check is the dose–response curve linking image sensitivity to accumulated snippet tokens, and then to the rate of invented bridges at the following step. We fit it on MMhops and Dyn-VQA, and reproduce it in a text-contamination form on SciAux by adding irrelevant auxiliary context to ScienceQA images.

The third check is the intervention table: the naive agent, periodic, random and triggered re-anchoring, snippet compression, and CAVR. We report MMhops accuracy, Dyn-VQA F1-Recall, and invented-bridge rate. INFOSEEK (single-hop, where MMhops-R1 reports 33.2%) serves as a control on which re-anchoring should neither help nor hurt much.

The fourth check is the demonstration-dose sweep and its task-vector counterpart. The hypothesis is supported if text demonstrations improve query formatting while lowering image sensitivity, and if patched vectors keep the formatting gain without the drop.

We accept the core claim if anchor replay recovers a non-trivial share of failures whose retrieval would otherwise be labeled missing. We reject it, and report the text-only diagnosis as sufficient for multimodal agents, if that share is negligible.