Teaching Multimodal Search Agents Where Their Reasoning Chains Break

A search agent that answers questions about images has to do several things in order. It looks at the picture, works out what is in it, writes a search query, reads what comes back, and then writes the next query from what it just learned. These agents are trained almost entirely on examples of chains that worked. They never see the exact step where a good chain turns bad, for example a query built around a logo the model only imagined, or a retrieved passage about the wrong entity that it accepts anyway. As a result they cannot tell a grounded step from a fluent but ungrounded one, and one early mistake carries through every later step. We propose to collect failed search chains, use a decomposing language-model judge to find the first step where a claim stops being supported by the image or the retrieved text, and train the agent on step-level preference pairs at that point. Each pair is two continuations that share everything before the break: one grounded, one not.

The point is practical. Today's best multimodal search agent is a prompting loop, and it still falls into what its authors call a "thinking trap": cascading hallucinations across search hops. Benchmarks also mostly score only the final answer, which hides cases where the answer is right but the chain behind it is invented. We therefore also propose a hop-level grounded accuracy metric, and we report how well the judge that produces our training labels actually agrees with humans, since that judge is itself a known source of error.

Core Claim

Our claim is that, for multimodal multi-hop retrieval, the informative negative signal sits at the first ungrounded hop. Preference pairs anchored there should reduce cascading hallucination more than three alternatives given the same budget of sampled trajectories: training on successful chains alone, preferences over whole trajectories, and preferences at randomly chosen steps.

The claim draws on several prior results. A theoretical and empirical study of multimodal process reward models, Training Data Efficiency in Multimodal Process Reward Models, shows that a step-scoring model learns its decision boundary almost only from "mixed" trajectories that contain the exact step where an error occurs. Their selection score keeps 10% of the data and matches training on all of it. That work stays inside closed visual math problems and labels steps with expensive Monte Carlo rollouts. We carry its central insight, that the error-onset step carries the gradient, into open-ended search. We replace rollouts with a claim-level semantic judge, and we train the agent itself rather than a separate verifier.

KnowPO applies preference optimization against two retrieval failures: ignoring good evidence and following distractors. It controls for length bias by rewriting candidates to similar lengths. But it works on whole answers over single documents, and its authors name multi-hop extension as open. We reuse its length-matching discipline at the step level.

Research Program

The agent. The agent follows the planner, retriever and sub-question-solver loop of OmniSearch from Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent. We instantiate it with an open vision-language backbone, Qwen2.5-VL-7B (one of the backbones used in the process-reward-model study), so its weights can be trained.

Sampling and decomposition. For each training question we sample several trajectories. A judge then breaks every hop into atomic claims. Some claims are about the image, such as "the jersey shows number 23". Others are about retrieved text, such as "the player was traded in 2019". The decomposition follows the parse-then-match design of VALOR-EVAL, which splits captions into objects, attributes and relations before matching them to ground truth. The judge checks each claim against two sources:

  • the image, using a vision-language judge;
  • the passages actually retrieved at that hop, using a text entailment check.

The first hop with an unsupported claim is marked as the break point.

Building pairs. At the break point we resample continuations from the shared prefix. A continuation the judge marks grounded becomes the preferred step, and the original ungrounded step becomes the rejected one. We rewrite the pair to roughly equal length, following KnowPO. Training uses a step-level preference loss of the direct-preference form:

$$ \mathcal{L} = -\log \sigma\!\left(\beta\left[\log\frac{\pi_\theta(s^+ \mid h)}{\pi_{\text{ref}}(s^+ \mid h)} - \log\frac{\pi_\theta(s^- \mid h)}{\pi_{\text{ref}}(s^- \mid h)}\right]\right) $$

The symbols are:

  • $h$ is the shared prefix: the image, the question and the hops before the break.
  • $s^+$ and $s^-$ are the grounded and ungrounded next hops, each consisting of a thought, a query and an extracted claim.
  • $\pi_\theta$ is the agent being trained.
  • $\pi_{\text{ref}}$ is the frozen starting agent.
  • $\beta$ controls how far the agent may drift from the reference.

The loss is standard. What is new is the choice of $h$ and $s^-$.

Where the image comes in. The multimodal part enters through which claims fail. Why LVLMs Are More Prone to Hallucinations in Longer Responses finds that attention moves from image tokens to the model's own generated text as outputs grow. MM-Snowball finds the same "visual fading" across dialogue turns, which explicit re-inspection of the image reverses. Both predict that image-claim failures will cluster at later hops, as retrieved text piles up. Our pairs therefore penalize visual neglect exactly where it happens. CAVR, the method proposed in MM-Snowball, and the decoding-time method Summary-Guided Decoding correct this only at inference time.

Training questions. We do not train on Dyn-VQA, which has only 1,452 questions and serves as our test set. Instead we build image-anchored versions of text bridge questions from HotpotQA, 2WikiMultihopQA and MuSiQue. Wherever the bridge entity can be shown visually, we replace its first mention with an image of it. This forces the first hop to be a perception step. We filter out cases where the image alone gives away the answer.

The judge as a measured component. We treat the judge as an instrument to be measured, not trusted. LLM judges are lenient toward plausible fabrications (D'Souza et al.), and they hallucinate on spatial and visual layouts (Chen et al.). We will hand-annotate break points on a held-out set of several hundred trajectories and report how well the judge localizes them. HaloQuest's Auto-Eval reports r = 0.99 agreement with humans, but on final answers to single-image questions (HaloQuest). We found no reported result on Lacuna for hop-level break-point localization by an LLM judge.

Two metrics. Final-answer F1-Recall on Dyn-VQA is the first. The second, hop-grounded accuracy, counts an answer as correct only if every hop passes the grounding check. It is motivated by Yan et al. and Zenz et al., who show that vision-language models often give the right answer with an explanation built on visual features that are not there.

Baselines

Naive reference points. The do-nothing reference is the untrained Qwen2.5-VL-7B in the same OmniSearch loop, plus the same backbone answering in one shot without retrieval. For context, the Dyn-VQA paper reports these F1-Recall scores:

System F1-Recall on Dyn-VQA
Base GPT-4V 30.25%
Bing Chat 31.44%
Gemini 37.69%
Heuristic multimodal retrieval pipelines about 37%
OmniSearch with GPT-4V (strongest existing system) 50.03%
Humans 55%

Strongest existing method. OmniSearch is the strongest reported system on this question. We cannot claim to beat a GPT-4V agent with a 7B model, so the fair comparison is OmniSearch run with our backbone and identical retrieval tools. We run it ourselves, and we also quote the GPT-4V number as a ceiling reference.

Rejection-sampling fine-tuning. The strongest trained competitor is rejection-sampling fine-tuning: supervised training on exactly those sampled trajectories whose final answer, and optionally every hop, the judge accepts. This is the positive-only recipe our thesis argues against. It gets the same sampled trajectory pool, the same judge, the same number of gradient steps, and the same learning-rate sweep (four values each).

Outcome-level preference baseline. This is KnowPO extended to whole trajectories: correct versus incorrect full chains, length-matched.

Ablations. Three ablations isolate the key ingredient:

  • Random-step pairs: preference pairs at a randomly chosen hop instead of the first ungrounded one.
  • Rollout-labeled pairs: break points found by Monte Carlo rollout estimates instead of the judge, at matched compute.
  • Text-only checking: only text claims are checked, which removes the image-grounding signal.

What counts as winning. All methods train from the same starting checkpoint. Beating the baselines means two things. First, hop-grounded accuracy on Dyn-VQA must exceed rejection-sampling fine-tuning and outcome-level preferences by more than the seed-to-seed spread across three seeds. Second, final-answer F1-Recall must not fall below the untrained OmniSearch-with-our-backbone baseline.

Why not just…?

Why not just fine-tune on the successful trajectories? This is the real threat. If filtering for chains that pass every hop's check is enough, then the negative signal adds nothing, and the cheaper method should be used. The process-reward-model study is indirect evidence that it falls short: their theory says trajectories that are uniformly correct give no gradient at the decision boundary. But that result concerns a verifier, not a generating agent. The matched-budget comparison against rejection-sampling fine-tuning settles it. If the two tie on hop-grounded accuracy, our core claim is refuted and we will report it that way.

Why not just use whole-trajectory preferences, as KnowPO does? With multi-hop chains, a rejected trajectory usually contains several correct hops before the break. Penalizing all of them blurs the signal and may teach the agent to distrust good first hops. KnowPO itself never tested multi-hop settings. The outcome-level baseline and the random-step ablation together show whether where the negative falls matters, not just whether negatives exist.

Why not just intervene at inference time, as CAVR does? MM-Snowball's V-shaped curve shows that simply prompting the model to look again recovers much of the lost accuracy. So a training-free fix might close the gap on its own. This is plausible enough that we include CAVR-style re-inspection prompting as an inference-time baseline, and also stack it on top of our trained agent. If prompting alone matches training, the honest conclusion is that visual fading in search loops is a decoding problem, not a learned bias. If the gains add up, the two methods address different failures.

Why not just label steps with Monte Carlo rollouts instead of a judge? Rollouts avoid judge leniency, but each search rollout requires live retrieval calls, which is far costlier than in closed-form math. The process-reward-model study also shows that rollout labels produce noisy pseudo-positives. The rollout-labeled ablation, at matched compute, together with the human-annotated localization audit, shows which labeling source gives better pairs per unit of cost.

Decisive Checks

Main comparison. Dyn-VQA is the primary test, scored on both F1-Recall and hop-grounded accuracy and broken down by its three categories. We expect the largest gain on the multi-hop category. If gains appear only on the rapidly-changing-answers category, the method is helping retrieval freshness, not reasoning.

Localization of the gain. Our mechanism predicts fewer later-hop image claims that are unsupported. We will plot ungrounded-claim rate against hop index for each method and test whether the trained agent's curve flattens where the baseline's rises.

Transfer. We run two transfer checks:

  • MM-Snowball, scored with its Snowballing Resistance Score. The paper gives no numeric SRS values, so we will report our own runs of every method.
  • HaloQuest's insufficient-context and false-premise questions. A good agent should abstain or search rather than invent. Auto-Eval's reported 68.1% for GPT-4o is an outside reference point.

Text-only sanity check. On MuSiQue, run text-only, the method should not underperform an IRCoT-style loop (Interleaving Retrieval with Chain-of-Thought Reasoning). That paper reports gains of 11–21 points in retrieval recall and up to 15 F1, relative not absolute.

Judge audit. If the judge localizes human-marked break points poorly, any downstream gains are suspect. We will then retrain with human-corrected labels on the audited subset and check whether the ranking of methods holds.