Grounding-Aware Expert Routing Across Hops in Multimodal Search Agents

Some questions about an image need several lookups. Take a photo of a building with the question "was the architect of this building older than the architect of the one in the second photo?" To answer it, a model has to recognize what is in each picture, search for facts, and then compare them. Current agents that do this run one dense network, the same weights, at every step. As text from searches piles up in the context, they appear to stop looking at the images. The strongest open agent of this kind, MMhops-R1, reaches 51.35% on its "bridging" questions, which chain from one image. It reaches only 22.01% on "comparison" questions, which must cross-reference several images. We propose to give such an agent a small set of expert adapters (swappable low-rank weight modules on a frozen backbone). A lightweight controller picks one adapter per reasoning step. Its decision depends on a measured signal: how much the model is currently attending to the images versus the retrieved text.

In two sentences: we hypothesize that multimodal multi-hop agents fail at later steps mainly because visual grounding decays as retrieved text accumulates. We test whether handing each step to an expert chosen from the live image-versus-text attention balance fixes this better than the obvious alternatives. Those alternatives are more dense capacity, routing by the action the agent is taking, and simply re-showing the image at every step.

Core Claim

The claim has two parts, and the second only matters if the first holds.

The first part is diagnostic. In a reinforcement-learning-trained multimodal search agent, the share of attention placed on image tokens falls with each hop. Wrong answers on comparison questions should be concentrated at hops where that share is low even though the question still depends on the image.

The second part is the method. Routing capacity at the level of whole hops fixes this decay better than three alternatives: a dense model of equal size, routing by explicit action type, and re-injecting the image. The controller is conditioned on a grounding signal and an evidence-source signal. It raises comparison accuracy without hurting bridging or single-hop accuracy.

For each hop $t$, the grounding signal is

$$g_t = \frac{A_t^{\text{img}}}{A_t^{\text{img}} + A_t^{\text{ret}}},$$

where $A_t^{\text{img}}$ and $A_t^{\text{ret}}$ are the total attention mass, averaged over heads in the upper half of the layers, that the tokens generated at hop $t$ place on image tokens and on retrieved-text tokens respectively. This is a white-box measurement of internal state, of the same kind DRAGIN uses to decide when to retrieve. We use it to decide which expert runs, not whether to retrieve.

How This Relates to Existing Work

Three papers supply the motivation.

MMhops-R1 trains a dense Qwen2.5-VL-7B agent with image search, text search and answer actions using DAPO, a group-based policy-gradient method. It leaves the comparison gap unexplained, and it spends the same capacity on every step. We reuse its environment, action space and training recipe, and change only how capacity is assigned.

R2-T2 shows that multimodal expert routers are "lazy." They default to language-aligned experts, which leaves an oracle gap of over 10%. R2-T2 repairs routing with test-time gradient steps on a fixed input, at 6–7x the FLOPs of a forward pass. We carry the lazy-router diagnosis into a setting where the input changes at every hop. We fix it with a trained controller rather than per-query optimization.

MATH-Vision reports that chain-of-thought can lower accuracy for some multimodal models because generation loses contact with the image. That is the same decay, observed within a single turn.

Two papers supply the mechanism.

PA-MoE routes LoRA experts per environment step, using a history-tracking router with a switching penalty. It reports an 8.1% drop when routing per token instead. That drop is the main reason we route per hop rather than per token. PA-MoE is text-only and routes on observation history. We add images, retrieval, and an explicit grounding input. We also ask whether learned phases beat the phases the action vocabulary already provides, a question PA-MoE did not face because its environments lack that structure.

Metis-HOME routes whole queries between a thinking and a non-thinking branch. It reaches a 46.1% average on six reasoning benchmarks and engages thinking on 98.1% of WeMath queries. Its decision is made once per query, and we test whether per-hop decisions matter.

Two papers constrain the design.

RoE shows that skipping whole layers in multimodal models is fragile: a 12% skip rate on LLaVA-HR cost 4.34%. This steers us toward adapters over a frozen backbone rather than skipping computation.

The entailment-tree MoE approach for multimodal multi-hop QA assigns experts statically to a retrieval head and a QA head. Its authors note that intermediate answers are underused across steps, which is the cross-step coupling we target.

Research Program

The work proceeds in three stages. The first stage can stop the project.

Stage 1: Diagnosis. We reproduce an MMhops-R1-style dense agent on Qwen2.5-VL-7B. We log $g_t$ at every hop of every trajectory on MMhops bridging and comparison splits, and annotate whether each hop's correct next action depends on image content. The prediction is threefold. $g_t$ should decline with hop index. Failed comparison trajectories should show low $g_t$ at image-dependent hops. Pushing attention back onto image tokens at those hops, for example by scaling attention logits toward image tokens, should recover some failures. If $g_t$ is flat, or does not separate successes from failures, the premise is wrong. In that case we report the diagnostic result and do not build the router.

Stage 2: Method. We attach $E=4$ LoRA experts to the frozen backbone. The number follows PA-MoE's scale. We treat it as a hyperparameter and do not tune it per task.

At the start of each hop, a small controller reads three inputs: the pooled hidden state of the previous hop, $g_{t-1}$, and the previous action type. It emits a distribution over experts. One expert runs for the whole hop.

Training is joint with the policy under the same DAPO outcome and format rewards. We add a switching penalty borrowed from PA-MoE. We also add a light auxiliary term that penalizes low $g_t$ on hops the controller sends to a designated "visual" expert. Without this term the experts could specialize arbitrarily; with it, one expert is encouraged to keep attending to the images. We ablate the term.

Stage 3: Analysis. We inspect what the experts learned. Do they split by evidence source (image versus retrieved text), by action, or by hop depth? We also measure whether routing to the visual expert actually raises $g_t$ and accuracy on image-dependent hops.

Baselines

The naive reference point is the base Qwen2.5-VL-7B prompted to use the same search tools without RL. External anchors come from MMhops-R1: GPT-4o at 36.62% on bridging and 8.76% on comparison, and Gemini-2.5-flash at 46.58% on bridging.

The strongest existing method is MMhops-R1 itself. The file reports 51.35% on bridging, 22.01% on comparison, and 33.2% on single-hop INFOSEEK. We cite these numbers, but our main comparison is against our own reproduction. The reproduction uses the same backbone, the same training data, the same number of RL steps and rollouts, and a LoRA of rank equal to the sum of our experts' ranks, so that trainable parameters match. It gets the same hyperparameter search budget as our method. Beating it means at least a 3-point gain on comparison accuracy that holds across three seeds, with bridging within one point of it and INFOSEEK not lower.

The ablations that isolate the key ingredient are all run under the identical budget:

  • Action-tag routing: one expert per action type, fixed by hand, no learned controller.
  • Per-query routing in the style of Metis-HOME: one expert chosen at hop zero and kept throughout.
  • Per-token routing: the same experts selected token by token, to recheck PA-MoE's fragmentation result in the multimodal setting.
  • Controller without $g_t$: our full method with the grounding input and auxiliary term removed.

Our method must beat action-tag routing and the no-$g_t$ controller on comparison. If it does not, the grounding signal is not what matters.

Why not just…?

Why not just add dense capacity or scale the model? A capacity-matched dense LoRA is exactly our main baseline, so this is settled directly. There is prior evidence that allocation can beat raw size: in PA-MoE, a 1.5B phase-routed model reached 93.8% on ALFWorld against 90.2% for a dense 7B model. That was text-only, though, so it motivates our hypothesis without confirming it. If the matched dense model ties us, our routing claim fails.

Why not just route by the action the agent is taking? This is the most serious threat. MMhops trajectories already label each hop as image search, text search or answer. Hard-coded per-action adapters are cheap and might capture most of the benefit. Unlike PA-MoE's environments, our setting provides phase labels for free. The action-tag baseline answers this directly. Stage 3 also checks whether learned experts split along action boundaries, which would favor this alternative, or along evidence-source boundaries within an action, for example two text-search hops, one of which must still consult the image. We commit to reporting it if action-tag routing wins.

Why not just re-show the image, or crops of it, at every hop? If grounding decay is the problem, putting image tokens back near the end of the context might solve it without any routing, at the cost of more tokens. This is also a real threat. We run it as an additional baseline, alone and combined with our router. If re-injection alone closes the comparison gap, the correct conclusion is that decay is a problem of attention position, not of capacity. We would report it that way. In that outcome our contribution is reduced to the diagnosis and the $g_t$ measurement.

Proposed Experiments

Main evaluation. This uses MMhops bridging and comparison splits (31,117 samples, 3–4 hops), scored by answer accuracy.

Secondary evaluations. INFOSEEK serves as a single-hop regression check. WebQA serves as a shallower multimodal multi-hop setting, where the entailment-tree method topped the leaderboard; the file gives no exact score, so no reported number is available. MATH-Vision measures whether the visual expert helps grounded reasoning without retrieval. The reference points there are GPT-4o at 30.39% and InternVL-Chat-V1-2-Plus at 16.97%, as reported by MATH-Vision.

Efficiency. We report wall-clock time per trajectory and the number of hops. Per-hop routing activates one adapter per hop, so it should add latency comparable to PA-MoE's reported 0.3%. We verify this rather than assume it.

Decisive checks. Four results decide the project:

  • Stage 1 must show that $g_t$ declines and predicts failure on image-dependent hops.
  • The full method must beat both the capacity-matched dense agent and action-tag routing on comparison accuracy across seeds.
  • Removing $g_t$ from the controller must erase most of the gain.
  • Image re-injection alone must fall short of the router.

Any single failure narrows the claim in a specific, reportable way.