Many questions about an image can only be answered through a chain of lookups. Take "Who founded the company that makes the watch the woman on the left is wearing?" The system first has to decide which woman, then which object on her is the watch, then identify its brand, and only then search for the founder. Current search agents run this chain as free text, with the model writing its own search queries. Once "the watch" silently becomes the other woman's bracelet, every later search is correctly executed on the wrong entity. The agent cannot tell that anything went wrong, so it loops. We propose to give such agents a small, explicit record of which region of which image each phrase refers to. We call it a referent ledger. Each later search must be built from ledger entries. When retrieved evidence contradicts an entry, the agent revises that one binding and re-runs only the steps that depended on it.
This sits between two literatures that have not met. Multimodal coreference resolution is the task of deciding when a phrase like "it" or "that building" and a region of an image denote the same thing. That literature has tools for keeping references stable across dialogue turns, but its systems never consult outside knowledge. The multi-hop reasoning literature has agents that search the web over several steps, but they treat images coarsely and have no notion of a persistent referent. The failures reported at the boundary look like coreference failures. These include the "thinking trap" in agentic multimodal search, and the collapse on questions that compare entities across several images. We want to test that diagnosis directly and fix it.
Core Claim
A large share of errors in multimodal multi-hop search comes from wrong or drifting bindings between phrases and image regions, not from retrieval or final reasoning. Passing bindings forward, as dialogue models already do, is not sufficient. The missing ingredient is revisability: each downstream hop records which bindings it depended on, and the agent can retract a binding once evidence contradicts it. Our prediction is concrete. On multi-image comparison questions and on images with several same-category instances, a ledger with revision should beat three alternatives, with a gap that grows with the number of candidate referents in the image:
- a ledger without revision;
- an agent that merely has a crop-and-zoom tool;
- the reinforcement-learned search agent it is built on.
What Prior Work Establishes
Evidence that motivates the proposal. OmniSearch, the planning agent introduced with the Dyn-VQA benchmark in Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent, loops over sub-questions and search tools. Its authors report two relevant failures. It breaks when "it" or "this person" must be tied to the right sub-region of an image across hops. And it fixates on a wrong early visual cue and cannot recover. MMhops-R1: Multimodal Multi-hop Reasoning trains a 7B Qwen2.5-VL agent with reinforcement learning to interleave thinking, image search, and Wikipedia search. It reaches 51.35% on single-image Bridging tasks but only 22.01% on multi-image Comparison tasks. Its image search operates on whole images, with no region-level tracking across steps.
Diagnostic work tells the same story at small scale. CLEVR-Dialog found accuracy up to 30% lower on questions needing coreference than on history-independent ones. Who are you referring to? Coreference resolution in image narrations shows that text-only resolution fails when an image contains several instances of the same category.
Methods we reuse. ChatterBox turns a grounded box into a feedback token for the next turn, which prevents pronoun drift within one image. It has no retrieval, and it never revisits a binding. Our ledger generalizes this token to many images and external evidence, and adds retraction.
The older "refer then find" design of Dual Attention Networks resolves the phrase in text before grounding it in pixels. Recursive Visual Attention walks back through history only when ambiguity is detected. We borrow both ideas: resolve the text first, then ground it, and walk back only on detected conflict.
SnapNTell shows that cropping to the entity before retrieval helps a fixed single-hop pipeline. We use its crop-then-link step as the grounding primitive inside each hop.
The state representation in MDST keeps static region features plus entity states updated each turn, and outperforms flat history concatenation on VisDial. We keep its separation of vision and entity state. Unlike MDST, we allow new regions to be grounded on demand rather than fixing the regions extracted at the start.
Negative results that constrain the design. According to Me found that entity and temporal graph memory did no better, and often worse, than a flat pile of items over multi-year personal data. The same paper found that key-value schema memory beat prose by about 20% under oracle retrieval.
This could be read as a strike against any structured memory. We read it as a strike against global, precomputed link structure built over noisy data. The ledger is the opposite in three ways: it is per-question, built on demand, and holds only a handful of entries, each with key-value fields in the style of that paper's schema memory. If the ledger nonetheless shows the same pattern, adding overhead without gains, that is informative, and the experiments are designed to reveal it.
Research Program
We build the ledger as a set of agent actions on top of the MMhops-R1 recipe. We keep the same backbone, the same preference-optimization training, and the same answer and format rewards, so that any difference comes from the actions available.
A ledger entry holds five fields:
- an image identifier;
- a box proposed by a grounding model from a resolved phrase;
- the linked entity name, obtained by cropping and searching;
- the evidence snippets that support the link;
- a confidence score.
Every search query the agent issues must reference at least one ledger entry. The system records which entries each hop consumed, so the reasoning chain carries explicit dependencies.
After each retrieval, a lightweight consistency check asks whether the new evidence is compatible with the attributes of the entries it depends on. For example, the retrieved brand might make watches, but the box might contain a bracelet. On conflict, the agent may call a revise action. Revise re-grounds the lowest-confidence upstream entry, choosing the next-best box or entity candidate, and invalidates only the hops downstream of it. This targeted repair replaces the circular re-querying seen in the thinking trap.
For multi-image comparison, the ledger also gives cross-image identity a concrete form. "The same building in both photos" becomes two entries that point to one entity. This is exactly the correspondence that the long-context models in DocHop-QA fail to maintain when they are simply given more page images.
The analysis side matters as much as the system. We will annotate about 500 MMhops items, split between Bridging and Comparison, with the gold image region for every intermediate entity. This lets each failure be attributed to one of three stages:
- a wrong initial binding;
- a correct binding followed by drift;
- correct bindings but a retrieval or reasoning error.
That attribution tests the core claim independently of any accuracy gains.
Baselines
Frozen backbone. The naive reference point is the Qwen2.5-VL 7B backbone answering directly from the image, with no tools. This shows how much any search helps.
Strongest existing method. The main competitor is MMhops-R1 itself. It is the strongest reported system on the benchmark closest to our question, at 51.35% Bridging and 22.01% Comparison accuracy in MMhops-R1. It also shares our backbone and training recipe, which isolates the contribution of the ledger. We will retrain it ourselves under identical conditions: same data, same number of optimization steps, same rollout budget, and the same cap on tool calls per question. We will report both our reproduction and the published numbers. Beating it means a statistically reliable gain on Comparison accuracy under a matched cap on tool calls, since revision could otherwise buy accuracy with extra searches. Bridging accuracy must not fall.
Dyn-VQA reference. On Dyn-VQA the strongest reported number is 50.03% F1-Recall for OmniSearch with GPT-4V in Benchmarking Multimodal Retrieval Augmented Generation. A 7B open model with a ledger is not directly comparable to a GPT-4V planner. We will therefore run OmniSearch's planning loop with our open backbone as the fair baseline, and cite the GPT-4V figure only as a ceiling.
Ablations isolating the key ingredient. Two ablations separate the parts of the ledger. The first is the ledger without the revise action. This is effectively ChatterBox-style referent feedback moved into a retrieval loop, which is the obvious combination and the one we must beat. The second is a crop-only agent: MMhops-R1 plus a grounding-and-crop action, but no persistent ledger. For single-image dialogue grounding, we report ChatterBox's T-score of 0.532 on CB-CoQ from ChatterBox, as a check that our grounding primitive is not weaker than the one we borrow.
Why not just…?
Why not just add a crop-and-zoom tool and let reinforcement learning figure out the rest? This is the most serious threat. SnapNTell's gains from cropping before retrieval suggest that much of the benefit may come from finer visual input alone. A crop tool, however, leaves no persistent object that later hops depend on, so it cannot support targeted retraction. It also gives two images no shared entity to point to. The crop-only ablation settles the question directly. If it matches the full ledger on Comparison and on the high-distractor subset, we will conclude that grounding granularity, not referent bookkeeping, was the bottleneck, and report that result.
Why not just put everything into a long-context model? DocHop-QA reports that adding page images often hurts relative to text alone. MMhops-R1 reports GPT-4o at 8.76% on Comparison. This suggests self-attention over a long concatenated context does not maintain cross-image identity. We include long-context prompting of the backbone with all retrieved text and images as a secondary baseline. The prediction is that its errors will concentrate in the "wrong binding" category of our attribution.
Why not just run text-only interleaved retrieval over a caption? IRCoT shows that iterative retrieval beats one-shot retrieval on text multi-hop benchmarks, by up to 15 F1. However, a caption cannot say which of two similar people or objects is meant, which is the failure documented in image-narration coreference. The heuristic caption-then-search pipeline also reached only about 37% F1-Recall on Dyn-VQA. We run caption-plus-IRCoT as a baseline and expect it to degrade fastest as the number of same-category instances grows.
Why not just build a knowledge graph of the scene and corpus upfront? The Linked Memory result in According to Me is direct evidence against this approach for noisy multimodal data. Precomputed region sets also cannot add regions that only become relevant mid-chain, as MDST's static features illustrate. We do not rerun this baseline at scale. We note it as the design the ledger deliberately avoids.
Proposed Experiments
Main comparisons. The primary evaluation is MMhops Bridging and Comparison accuracy for all systems under matched tool-call caps, with Dyn-VQA F1-Recall as a second benchmark.
Decisive check 1: distractor-controlled evaluation. We bucket the annotated MMhops subset by the number of same-category instances in the relevant image. The claim predicts that the gap between the full ledger and the crop-only and no-revision ablations widens with that count. A flat gap would falsify the claim that referent identity is what matters.
Decisive check 2: error attribution. Using the gold intermediate regions, we measure what fraction of MMhops-R1's failures begin with a wrong or drifting binding. If that fraction is small, the premise is wrong, however the accuracy numbers come out.
Decisive check 3: recovery. We inject a wrong first-hop binding by swapping in a same-category distractor box. We then measure how often each system recovers the correct answer within the tool budget. Only the revise action is designed for this, so it is the cleanest test of revisability.
Sanity and cost checks. We run CB-CoQ to confirm the grounding module is competitive. We also report the average number of tool calls and wall-clock latency, since MMhops-R1 already flags latency as a limitation and revision adds calls.
References
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent
- MMhops-R1: Multimodal Multi-hop Reasoning
- CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
- Who are you referring to? Coreference resolution in image narrations
- ChatterBox: Multimodal Referring and Grounding with Chain-of-Questions
- Dual Attention Networks for Visual Reference Resolution in Visual Dialog
- Recursive Visual Attention in Visual Dialog
- SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM
- Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations
- According to Me: Long-Term Personalized Referential Memory QA
- DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions