Looking Back at the Image: Revising Visual Guesses with Downstream Evidence in Multimodal Multi-Hop Reasoning

Consider the question "What year was the team whose logo appears in this photo founded?" A vision-language model must first decide which team the logo belongs to, then look up that team, then read off the year. When models fail at questions like this, the cause is usually not the lookup. It is the first step. The model misreads a logo or a jersey pattern, commits to the wrong team, and every later step reasons fluently from that wrong premise. Piecing It All Together, which introduced the MMCV multi-hop claim-verification benchmark, identifies this kind of visual misidentification as the dominant root cause of fabricated reasoning chains. It also reports a troubling pattern: as questions go from one to four reasoning steps ("hops"), models become more confident even as accuracy falls, trailing humans by about 27% at four hops.

We propose to treat the first visual identification as an open hypothesis rather than a committed fact. The system keeps several candidate identities for each image region and follows each one forward through retrieval. It then uses what it retrieves to go back and re-examine the image. If the retrieved text says the candidate team plays in red and white vertical stripes, the system asks the image whether the region shows red and white vertical stripes. Candidates whose retrieved facts contradict the pixels lose weight. The answer and its confidence come from whichever identities survive. In two sentences: later retrieval steps produce concrete, checkable predictions about what the image should look like, and checking those predictions against the image corrects the early perceptual errors that drive multimodal hallucination. The same mechanism yields a confidence score that falls when the visual evidence is ambiguous, which is the opposite of the overconfidence-at-depth pattern.

Core Claim

Multi-hop retrieval is currently used only in the forward direction: perception feeds retrieval. We claim that downstream textual evidence is an underused signal for perception. Coupling each candidate visual identity to the facts retrieved about it, and then checking those facts against the image, will do three things. It will recover a substantial share of hop-one misidentifications. It will raise end-task accuracy on bridging-style multimodal multi-hop questions. It will make confidence decrease, rather than increase, with reasoning depth when evidence is weak.

The structure that makes this work is a small heterogeneous graph built per question at inference time. It contains region nodes (detected image regions), candidate-entity nodes attached to each region, fact nodes retrieved for each candidate, and "prediction" edges that run from fact nodes back to region nodes. Each prediction edge is labeled with a visually checkable attribute. Scoring is a simple belief update. For region $r$ with candidate entity $e$, let $P_0(e \mid r)$ be the model's initial identification probability. Let $c_1,\dots,c_m$ be the checkable attributes extracted from facts retrieved about $e$. Let $q(c_j \mid r)$ be the probability, estimated by a visual question to the model, that attribute $c_j$ is visible in $r$. Then

$$P(e \mid r, \text{evidence}) \propto P_0(e \mid r)\prod_{j=1}^{m} q(c_j \mid r)^{\lambda},$$

where $\lambda$ is a single tempering weight tuned on a development split. When several regions in the same image must be mutually consistent (a stadium and a team, a landmark and a city), the joint score multiplies across regions and keeps only compatible assignments. The final answer is the mass-weighted vote over surviving hypotheses. That mass is also the reported confidence, and the system abstains below a threshold.

Relation to Prior Work

Several systems already sit at this intersection, and the proposal is defined by how it differs from each.

M3KG-RAG builds an offline multimodal knowledge graph from video and audio and prunes retrieved candidates with GroundingDINO, an open-vocabulary object detector. Its gain on VALOR (a model-as-judge score rising from 32.42 to 44.67) motivates grounding checks. Its check, however, asks whether an object exists in the scene. It does not ask whether downstream facts about a specific identity agree with the pixels, and it runs once, before reasoning.

MMAPG grows a planning graph of sub-questions and evidence and backtracks at dead ends. It reaches 70.6 F1 on MultimodalQA and 65.9 accuracy on WebQA. It backtracks when retrieval fails, not when retrieval succeeds but contradicts the image. A confidently wrong identity produces no dead end, so MMAPG never revisits it.

BRIT traverses a unified image-text graph in both directions using Prize-Collecting Steiner Tree optimization, a method for selecting a compact connected subgraph that covers high-value nodes. Its "bidirectional" traversal is a retrieval strategy over a static graph. It is not a verification loop over perceptual hypotheses.

MMhops-R1 trains an agent with flat image-search and text-search tools. It is the strongest reported system on MMhops, but it keeps one trajectory and no structured record of which visual claim each later step depends on.

Process-supervision work supplies a contrasting design rather than a method we reuse. GRAT uses tree search with step rewards on text multi-hop QA. Adversarial Training for Process Reward Models trains step verifiers adversarially. Both learn a verifier that judges steps in the text stream. Our check is training-free and routes the judgment back to the image, which is where MMCV locates the errors.

MR-MKG finds that text-only knowledge graphs underperform multimodal ones because captions lose visual detail. That result constrains us: re-checking retrieved facts against a caption, rather than against the image, should be the weaker ablation.

Research Program

The system has four stages, all run with one open vision-language model (Qwen2.5-VL-7B, matching MMhops-R1's backbone) so that comparisons isolate the method rather than the model.

Stage 1: candidate generation. A detector proposes regions that the question depends on. The model produces top-$k$ candidate identities per region with probabilities, using constrained decoding, optical character recognition for any visible text, and retrieval of reference images by visual similarity. We report top-$k$ recall of the gold identity, because the method cannot recover an identity it never proposes.

Stage 2: forward retrieval per candidate. Each candidate runs the same iterative retrieval loop used by the baselines, in the style of IRCoT, producing fact nodes.

Stage 3: extracting visually checkable predictions. A prompted extractor rewrites retrieved facts into attributes that could in principle be seen: colors, printed text, counts, shapes, and spatial arrangement. Facts with no visual consequence (such as founding year) are marked uncheckable and contribute nothing.

Stage 4: back-checking and scoring. Each attribute is posed as a targeted visual question on the cropped region. Answers are converted to probabilities from the model's yes/no token likelihoods. To counter the known bias of such models toward answering "yes," every check is paired with a contrastive negation, and each "yes" probability is normalized against the probability for the negated attribute.

The analysis component annotates 500 MMhops bridging questions and 500 MMCV claims with the gold hop-one identity. This lets every error be attributed to perception or to later reasoning, and lets us count how many perception errors the back-check flips.

Baselines

The naive reference point is the same backbone answering directly with single-pass retrieval over the question. This shows how much any multi-hop machinery helps at all. On MMhops, the file reports vanilla GPT-4o at 36.62% on bridging tasks and 8.76% on comparison tasks (MMhops-R1). We will also report the direct-prompting number for our own backbone.

The strongest existing method on our primary benchmark is MMhops-R1: a Qwen2.5-VL-7B agent with image and text search, trained with DAPO (a reinforcement-learning method that optimizes the policy against reward signals). It reports 51.35% accuracy on bridging tasks and 22.01% on comparison tasks. We will run its released recipe on the same backbone and search corpus. Beating it means exceeding 51.35% on bridging by a margin larger than seed variance, at no more than twice its average model calls per question. We apply the same back-check to MMhops-R1's single trajectory as a drop-in module to test whether the gain composes with its trained policy.

On MMCV, the strongest reported figure is Gemini 1.5 Flash open-book at 70.92 average F1 (Piecing It All Together). The file gives no per-hop numbers for GPT-4o or LLaVA-v1.5-7B. Because our backbone is smaller, the fair target is the gap to that figure at three and four hops, together with calibration.

On MultimodalQA and WebQA, MMAPG is the strongest cited system, at 70.6 F1 and 65.9 accuracy respectively. We rerun it with our backbone because its reported numbers use different models.

We also run a matched-compute Best-of-N baseline: N independently sampled trajectories with majority vote, where N is set so that total model calls equal ours. This is the most dangerous competitor (see below).

The key ablations isolate the back-check itself. The first keeps $k$ hypotheses but scores them by $P_0$ alone. The second uses $k=1$ with back-checking, which can abstain but not replace the identity. The third checks retrieved facts against a generated caption instead of the image. The fourth checks randomly chosen attributes rather than retrieval-derived ones.

All systems share the same retrieval corpus and backbone and receive the same development-set tuning budget of 20 configurations each. We report model calls and wall-clock time per question.

Why not just…?

Why not just sample many chains and take a majority vote? Best-of-N sampling is what CoRAG uses to scale test-time compute, and it does keep multiple hypotheses alive. This is a real threat. Our argument is that samples from one model share its perceptual bias: if the model reads a logo as team A 80% of the time, voting entrenches team A. Sampling also never consults the image a second time with a targeted question. The matched-compute Best-of-N baseline settles this directly. If it matches our accuracy and calibration on the annotated perception-error subset, the back-check adds nothing, and we will report that.

Why not just verify objects with a grounding detector before reasoning, as M3KG-RAG does? Detector-based pruning confirms that a logo or a jersey is present, but not whose it is. The errors MMCV documents are identity errors on objects that are genuinely present. The $k=1$ ablation with a GroundingDINO-style existence check measures this. We expect it to reduce hallucinations on absent objects but leave identity errors unchanged.

Why not just use a bigger model? The file shows scale does not close the gap. GPT-4o reaches 8.76% on MMhops comparison tasks, below a 7B agent, and frontier models trail humans by about 27% at four hops on MMCV. We will still run the back-check on top of a larger model to test whether the benefit shrinks with stronger perception. That trend is itself informative.

Why not just train a multimodal step verifier? An adversarially trained process reward model is a heavier, learned alternative (APRM). No multimodal version exists in the file, and training one needs step labels we do not have. We treat it as complementary: if the back-check works, its pass/fail signals are a natural source of step labels for such a verifier.

Proposed Experiments and Decisive Checks

The primary evaluation is MMhops bridging and comparison accuracy, along with MMCV F1 broken down by hop count (one to four). The secondary evaluation is MultimodalQA cross-modal F1 and WebQA accuracy. Calibration is measured by expected calibration error, the average gap between stated confidence and actual accuracy, reported per hop count. We also report selective accuracy at fixed coverage, meaning accuracy on the questions the system chooses to answer at a set answer rate.

Three results would confirm the claim. First, on the annotated subsets, at least a third of hop-one misidentifications that fall within top-$k$ recall are flipped to the correct identity. Nearly all of this gain should disappear in the caption-check and random-attribute ablations. Second, confidence decreases with hop count on MMCV, reversing the positive slope reported there, while matched-compute Best-of-N retains a flat or positive slope. Third, end accuracy on MMhops bridging exceeds 51.35% at comparable compute.

The failure modes are measurable. Top-$k$ recall may be too low, in which case the ceiling is perception rather than verification. Retrieved facts may rarely yield visible attributes, which the fraction of uncheckable predictions will reveal. Yes-bias may survive the contrastive normalization, in which case back-checks will pass wrong candidates at a rate we can estimate on synthetic mismatched pairs. Each of these would tell us precisely where the idea breaks, and each would be a reportable result.