Some questions can only be answered by chaining facts, and the first link comes from a picture. "What year was the architect of the building on the left born?" requires spotting the correct building, naming it, looking up its architect, and then looking up the birth year. When a system gets such a question wrong, the error usually starts at one step and then carries through every later step. Today's evaluation cannot say which step failed. Final-answer accuracy says nothing about the chain. Holistic judges read the whole trajectory, which here means the full recorded sequence of reasoning steps, searches, and retrieved passages. Image-caption metrics are known to catch wrong objects but miss wrong relations, such as "left of" versus "right of" or "holding" versus "next to."
We propose to break each reasoning trajectory into small, individually checkable claims, label each claim by the evidence that should support it (a region of the query image or a specific retrieved passage), and check each claim only against that evidence. The same claim-level verifier serves two purposes. As a diagnostic, it shows at which hop and in which modality multimodal multi-hop systems actually fail. As an inference-time signal, it can rerank sampled trajectories or restart reasoning at the first hop that fails. The central bet is that routing each claim to its own evidence beats a single judge that sees everything at once. That bet can be falsified with a single ablation.
Core Claim
Existing systems at this intersection either verify retrieved items or train the whole chain end to end, and none checks the model's own intermediate claims against the specific evidence that should support them. M3KG-RAG builds a multimodal knowledge graph and uses an open-vocabulary object detector (GroundingDINO) to prune retrieved entities that do not appear in the video. That check applies to knowledge-graph entries fetched from storage, and it tests only whether an object is present, not whether a relation holds. MMhops-R1 trains a 7B agent with reinforcement learning to call image and text search tools, but nothing checks its intermediate steps. It reaches 51.35% on MMhops Bridging and 22.01% on Comparison. EntRAG improves how the image entity is matched and reranked for knowledge-based visual question answering, but it runs a fixed pipeline with no way to backtrack.
Our claim is that a large share of the remaining errors are wrong visual links: a misidentified bridge entity, a wrong attribute, or a wrong spatial relation. We further claim that a verifier which checks visual claims against image crops, and textual claims against the cited passage, detects these errors much better than holistic multimodal judges or presence-only detectors. Detecting them early should turn into accuracy gains when the inference budget is held fixed.
The motivation comes from the visual-grounding evaluation literature. Cornia et al. survey caption metrics and find that even the best ones catch missing or invented objects but systematically miss invented relations. CLIPScore, which scores a caption by its similarity to the image in CLIP's shared image-text embedding space, reaches 92.6% on FOIL noun-swap detection in its reference-based variant. Noun swaps are exactly the easy case. We take this blind spot and move it inside a multi-hop chain, where one wrong relation silently changes the entity that every later search depends on.
Research Program
The first component is a claim decomposition and routing step. Each trajectory from a multi-hop agent is rewritten into atomic claims, such as "the object at the left is a Gothic cathedral," "this cathedral is Cologne Cathedral," or "Cologne Cathedral's architect was Gerhard." A language model tags each claim as visual or textual. Visual claims are tied to an image region and textual claims to a specific passage. Visual claims are further split into three types: identity, attribute, and relation.
The second component is the verifier itself. Visual claims go to a Qwen2.5-VL-7B model that sees only the relevant crop, or, for relation claims, the union of the two regions, and returns a probability that the claim is supported. We use the crop idea from iVGR, which reports that a test-time crop tool adds 3.5 points on HR4K. Textual claims go to the same model in an entailment-style mode against the cited passage alone.
We train the verifier on perturbed trajectories, following FOIL's recipe of introducing controlled errors. Starting from correct MMhops trajectories, we swap the bridge entity for a visually similar neighbor, flip spatial relations, change attributes, and substitute passages about near-miss entities. The model learns to reject these swaps. The swap types are chosen to cover the relational errors that Cornia et al. identify as the persistent blind spot.
The third component uses the verifier at inference time. A trajectory's score is the minimum support probability over its claims. The minimum is used because one failed link is enough to invalidate a chain.
We compare two uses of this score at a fixed number of model calls. In best-of-K reranking, K trajectories are sampled and the highest-scoring one is kept. In first-failure restart, the agent resamples from just before the lowest-scoring claim. First-failure restart is a deliberately cheap alternative to the Monte Carlo tree search in GRAT, which searches a tree of reasoning branches guided by per-step reward scores on text-only multi-hop QA. We choose the cheaper method because iterative agents already suffer from growing token cost and latency, as the IRCoT and GenGround papers document.
The fourth component is a human-annotated diagnostic set. We sample about 1,000 trajectories from an MMhops-R1-style agent on MMhops, and a smaller set from WebQA. Annotators mark each claim as supported or unsupported and identify the first failing hop. This set is the ground truth for evaluating the verifier, and it also measures directly how errors are distributed across hops and modalities.
Baselines
The do-nothing reference is a single greedy trajectory from the agent itself. We reproduce MMhops-R1's recipe on Qwen2.5-VL-7B, or use its checkpoint if one is available. The reported figures to beat come from MMhops-R1: 51.35% accuracy on MMhops Bridging and 22.01% on Comparison, against 36.62% and 8.76% for GPT-4o. Our reproduction must land within a few points of these figures before any comparison counts.
The strongest cheap competitor is self-consistency: sample K trajectories from the same agent and take a majority vote on the final answer. It uses the same samples, and therefore the same budget, as best-of-K. We treat it as the main threat. Beating it means a statistically significant accuracy gain at matched K on both MMhops splits, with the gap expected to be largest on Comparison, where the multi-image evidence gives more room for wrong visual links.
The strongest existing verifier baseline is a holistic multimodal judge. We use Prometheus-Vision, a LLaVA-based judge that writes feedback and then assigns a 1–5 score, which reached a Pearson correlation of 0.786 with humans on LLaVA-Bench. To make the comparison fair, we also fine-tune a Qwen2.5-VL-7B judge that sees the whole trajectory with the full image and all passages. It is trained on exactly the same perturbed data as our verifier and gets the same tuning budget.
The detector-pruning baseline follows M3KG-RAG: GroundingDINO checks that every named visual entity is present, and trajectories that fail are rejected. This baseline isolates whether checking relations and attributes adds anything beyond checking presence.
Two ablations isolate the key ingredient. The first, a caption-routed verifier, checks visual claims against a generated caption of the crop instead of the pixels. The second, an unrouted claim verifier, still scores each claim separately but sees all the evidence for every claim.
On the diagnostic set, the verifier comparison is measured by per-claim AUROC (area under the curve of true versus false positive rates, where 0.5 is chance) and by first-failure localization accuracy. The claimed ingredient holds only if routing to pixels beats both ablations, with the clearest gain on relation claims.
Why not just…?
Why not just use self-consistency with more samples? Majority voting helps when mistakes are scattered across different wrong answers. It fails when a similar-looking wrong entity is consistently preferred, because then most samples converge on the same wrong answer. Our diagnostic set measures directly how often the majority is wrong in exactly this way. If this rarely happens, self-consistency will match us and the inference-time claim fails, while the diagnostic finding still stands. We run this comparison at matched K for that reason.
Why not just caption the image and run a text multi-hop method? WebQA found that reducing images to text tags or captions loses the visual detail that multi-hop questions depend on, and its best fine-tuned systems scored about 24% combined accuracy and fluency against more than 94% for humans. That result concerns answering, not verification, so we test it again with the caption-routed ablation. If captions verify relation claims as well as crops do, the pixel-routing part of the proposal is unnecessary.
Why not just ask a strong multimodal judge to grade the whole trajectory? A holistic judge has to find the one wrong link inside a long transcript. Prometheus-Vision's own paper reports that such judges favor longer outputs and are sensitive to prompt wording. Cornia et al. report that learned metrics miss relational errors even for single captions. Still, a fine-tuned whole-trajectory judge trained on our perturbations is a real threat. The unrouted-judge baseline with identical training data settles whether routing, rather than the training data, produces the gain.
Why not just use tree search as in GRAT? GRAT's tree search improved Llama3-8B on 2WikiMultiHopQA from 60.6% to 66.8%, but only after self-training on 11,000 trajectories, and it needs a per-step reward model that no one has built for visual steps. Our verifier is that missing per-step signal. We keep search simple so that the effect of the signal is not confused with the effect of the search method. A tree search driven by our verifier is a natural follow-up if the verifier proves itself.
Proposed Experiments
The first experiment is the error audit. On the annotated MMhops and WebQA trajectories, we report where the first failing claim occurs, broken down by hop position and by claim type: visual identity, visual attribute, visual relation, and textual. This check decides whether the project continues. If fewer than roughly a third of failures start at visual claims, pixel-level verification cannot move final accuracy much, and we will report the audit as the main contribution.
The second experiment evaluates the verifier. On the same set, we compare per-claim AUROC and first-failure localization accuracy for our verifier, the holistic judge, the detector-pruning check, and the two ablations. As an external sanity check on noun-level detection, we also report FOIL accuracy, where RefCLIP-S scores 92.6% according to CLIPScore.
The third experiment measures end-to-end accuracy at matched compute. We report MMhops Bridging and Comparison accuracy for greedy decoding, self-consistency, holistic-judge reranking, detector pruning, best-of-K with our verifier, and first-failure restart, at K of 4, 8, and 16, with total generated tokens reported alongside accuracy.
For transfer, we evaluate zero-shot on WebQA, where no reported result for a verifier-reranked agent was found on Lacuna, and on InfoSeek. On InfoSeek, EntRAG reports 44.5 and MMhops-R1 reports 33.2%. InfoSeek is essentially single-hop, so gains there should be small. If they are not small, that would suggest the verifier is acting as a better entity identifier rather than a chain checker, and we would analyze it on those terms.
References
- M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation
- MMhops-R1: Multimodal Multi-hop Reasoning
- EntRAG: Entity-Centric Retrieval-Augmented Generation for Knowledge-based Visual Question Answering
- Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- iVGR: Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answering
- Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
- WebQA: Multihop and Multimodal QA