Vision-language models now write image descriptions that go well past naming what is visible. A description might say "the runner wearing the medal won the marathon held in the stadium behind her, which opened in 1992." Whether that sentence is true depends on a chain of facts. Some links are visual: there is a runner, she wears a medal, a stadium is in the background. Others need outside knowledge: which stadium it is, and when it opened. Automatic metrics that score such descriptions without human-written reference captions ("reference-free" metrics) either compare one global image embedding to one text embedding, or check isolated noun phrases one at a time. Neither follows the chain. The first is known to miss negation and word order. The second loses the links between facts.
We propose a reference-free evaluator that treats a description as a chain of dependent claims. Any entity the description leaves unnamed ("the stadium behind her") is resolved from the image and from retrieved text, not taken from the description's own wording. That resolved entity is then carried into the next check. The evaluator outputs a score and also the specific link where the description first goes wrong. Our bet rests on an asymmetry: checking a stated chain should be much easier than answering the corresponding question from scratch. If so, reliable evaluation of multi-hop descriptions is possible even where current multi-hop answering agents remain weak.
Core Claim
The strongest multimodal multi-hop answering agent in the file, MMhops-R1, is a Qwen2.5-VL-7B agent trained with reinforcement learning to alternate between image search, text search, and answering. It reaches only 51.35% accuracy on MMhops bridging questions and 22.01% on comparison questions. Building an evaluator by asking such an agent to re-derive each fact would inherit those error rates.
We claim that verification under hypotheses supplied by the candidate description is a different and easier problem, for two reasons. The description tells the verifier which entity to look for at each step, which narrows both the visual search and the text retrieval. And a wrong hypothesis usually produces a visible contradiction rather than a silent wrong answer.
The specific new ingredient is evidence-resolved bridge threading. At every link, the verifier resolves the referent from evidence and passes that resolved identity forward. It does not substitute whatever entity the description asserts. This is what separates our method from atomic claim decomposition. Atomic decomposition checks "Stadium X opened in 1992" if the description names Stadium X. It cannot check whether the stadium in the image is Stadium X, or whether the medal-wearer and the race-winner are the same person.
Research Program
The work has three parts: a benchmark, a verifier, and an analysis of when the verifier's own errors cascade.
Benchmark. No benchmark in the file measures evaluation of multi-hop descriptions, so we build one. MMhops supplies images paired with 3–4 hop bridging and comparison questions and gold answers. We convert each question and answer into a declarative description, following the VQA-to-caption conversion used by Ahmadi et al. for their hard-negative benchmark. We then create corrupted variants by perturbing exactly one link of three kinds:
- Visual link: swap the identified visual entity, attribute, or relation.
- Knowledge link: keep the visual entity correct but attach a wrong external fact.
- Join: keep both visual entities and their facts correct but assert a relation between them that does not hold.
Because each negative differs from its positive at one known link, the benchmark carries hop-level labels for free. The file does not say whether MMhops releases intermediate hop annotations. If it does not, we reconstruct chains automatically and have annotators verify a subset. We also collect human quality judgments on a sample of real model-generated descriptions of MMhops images, so that the benchmark is not purely synthetic.
Verifier. The verifier is built on the same Qwen2.5-VL-7B backbone as MMhops-R1 so that comparisons isolate method rather than model size. A parsing step rewrites the description into an ordered chain. Each claim refers to earlier entities through variables, and each claim is tagged as visual or knowledge. Each claim then gets a check matched to its tag:
- Visual claims: crop and zoom on candidate regions to test grounding.
- Knowledge claims: retrieve text conditioned on the evidence-resolved entity, in the interleaved style of IRCoT.
Each check returns supported, contradicted, or unverifiable. Negated claims are checked as negations, which directly targets the negation blindness that Ahmadi et al. document. The final score aggregates verdicts along the chain, and the first contradicted link is reported.
Cascade analysis. The evaluator can fall into the failure it is meant to catch. If it misresolves the first bridge entity, it may reject a correct description or accept a wrong one. The RAG attribution survey and CoRAG describe exactly this cascading in generation. We therefore measure false-rejection rates on correct descriptions as a function of chain length. We also test a simple safeguard: when hop-1 resolution confidence is low, the verifier keeps the top two candidate referents and verifies forward under both.
Baselines
Naive reference point. CLIPScore is the naive point. It is the cosine similarity between CLIP image and text embeddings, computed with no learning on our data. On the hard-negative benchmark of Ahmadi et al., CLIPScore, UMIC, and PAC-S fail to rank captions correctly in about 46% of cases, against a 15.4% human failure rate. We expect similar or worse behavior on knowledge and join perturbations. Any method that does not clearly beat it has failed.
One-pass judges. The strongest one-pass judges are:
- G-VEval: prompts GPT-4o with chain-of-thought and reads a continuous score from token probabilities. It reports a Kendall tau-b of 60.385 on Flickr8k-Expert in reference-free use, the highest correlation figure in the file.
- Prometheus-Vision: a LLaVA judge trained on rubric feedback. It reports a 0.786 Pearson correlation with humans on LLaVA-Bench.
- VisCE2: extracts a static visual inventory before judging. No reported result was found on Lacuna for VisCE2 as a number.
We run G-VEval exactly as published with GPT-4o. We also re-run its prompt on our Qwen2.5-VL-7B backbone. To equalize test-time compute, the one-pass judge gets Best-of-N sampling with N chosen so that its total generated tokens match our verifier's average.
Atomic decomposition with a strong verifier. This is a modern version of QACE, which generates questions from noun phrases and answers them from the image. QACE's Visual-T5 answerer is replaced by the same Qwen2.5-VL backbone with the same crop and retrieval tools. Claims are split into atomic facts that use entities as named in the description.
Answer-then-compare. We generate the multi-hop question implied by the description, let MMhops-R1 answer it, and compare its answer to the description's assertion.
Key ablation. Our full verifier is run with bridge entities taken from the description's text rather than resolved from evidence. Everything else is held fixed.
What counts as beating them. On the hop-perturbed benchmark, success means higher pairwise ranking accuracy than every baseline, with the margin concentrated on join and knowledge-link negatives. On human-judged real descriptions, success means higher Kendall correlation. On standard benchmarks (Flickr8k-Expert, Pascal-50S, Composite), success means no loss larger than a couple of tau points relative to the one-pass judge on the same backbone.
Why not just…?
Why not just prompt GPT-4o to judge the description directly? Zero-shot GPT-4o answers only 36.62% of MMhops bridging and 8.76% of comparison questions (MMhops-R1). Those numbers suggest it cannot silently verify chains whose facts it cannot produce. Answering accuracy is not judging accuracy, though, and the asymmetry we rely on would also help a direct judge. This is a real threat. The compute-matched G-VEval comparison on join and knowledge-link negatives settles it. If GPT-4o matches us there, the contribution shrinks to the benchmark.
Why not just extract a richer static visual inventory first, as VisCE2 does? VisIT-Bench shows that generic pre-extracted descriptions support only 31% evaluation success, against 91% when the descriptions are written for the specific instruction. A static inventory cannot know which background object will become the bridge to an outside fact. Knowledge-link negatives will be undetectable for any method without retrieval. We will report VisCE2 separately on visual-link negatives, where it may be competitive.
Why not just decompose into atomic claims and check each with a strong VLM? This is the most serious alternative. Many relational claims can be phrased atomically. Its weakness is predicted by Decomposition Dilemmas, which finds that splitting a claim strips the connective tissue between facts. QACE's own degradation on Flickr8k from unanswerable fragment questions points the same way. Still, the key ablation decides it. If evidence-resolved threading gives no gain over text-substituted bridges on join and implicit-bridge negatives, we will conclude that a well-tooled atomic checker suffices and report that.
Why not just reuse MMhops-R1 as the checker? Answering the implied question is bounded by MMhops-R1's 51.35% bridging accuracy. The answer-then-compare baseline tests directly whether conditioning on the stated chain beats re-deriving it.
Proposed Experiments and Decisive Checks
The central experiment is pairwise ranking accuracy on the hop-perturbed MMhops benchmark, reported separately for visual, knowledge, and join perturbations and for 3- versus 4-hop chains. The two results that most directly test the proposal are:
- the gap between our verifier and atomic decomposition on join negatives;
- the gap between our verifier and answer-then-compare overall.
Together these test the threading ingredient and the verification-versus-answering asymmetry.
The second experiment measures attribution. For negatives the method correctly rejects, we check how often its reported first failing link matches the perturbed link. No baseline in the file produces hop-level attribution, so chance and a "blame the last link" heuristic serve as reference points.
The third experiment is the cascade analysis: false-rejection rate on correct descriptions as chain length grows, with and without the two-candidate safeguard.
Finally, we check that the method still works on ordinary captions. We report Kendall correlation on Flickr8k-Expert against G-VEval's 60.385, and accuracy on Pascal-50S and Ahmadi's negation and word-order perturbations.
The proposal fails cleanly in either of two cases. One is if the text-substituted ablation matches the full verifier. The other is if a compute-matched one-pass judge matches it on join negatives.
References
- MMhops-R1: Multimodal Multi-hop Reasoning
- An Examination of the Robustness of Reference-Free Image Captioning Evaluation Metrics
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey
- Chain-of-Retrieval Augmented Generation
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o
- Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
- Vision Language Model Based Caption Evaluation Method Leveraging Visual Context Extraction
- QACE: Asking Questions to Evaluate an Image Caption
- VisIT-Bench: A Dynamic Benchmark for Evaluating Instruction-Following Vision-and-Language Models
- Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?