Executable Visual Checks for Retrieved Knowledge

When a vision-language model answers a question like "what year was this bridge finished?", it has to name the bridge before it can look anything up. That naming step is where these systems mostly fail. On the INFOSEEK benchmark, giving a retrieval pipeline the correct entity lifts accuracy from 18.2% to 45.6% (Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?). Retrieval then adds a second failure: plausible passages about the wrong entity, or about the right entity with wrong facts, pull the reader off course. Reasoning models are especially exposed, because step-by-step thinking tends to justify a misleading premise rather than reject it (Thinking in a Crowd).

We propose to treat the model's guess about which entity is in the image as a hypothesis the system can test. The system writes a small program that commits to a candidate entity and retrieves its article. It then turns a few distinguishing claims from that article into checks that vision tools can run on the image, for example "a suspension bridge with two red towers" becomes a detector call and a color query. If the checks fail, the program rolls back and tries the next candidate, and if no candidate survives, it declines to answer. The iterative refinement loop, the part that revises and re-runs the program, is aimed at the retrieval step rather than at code bugs. What drives each revision is agreement between retrieved text and pixels, not a stack trace.

Core Claim

Knowledge-based visual question answering fails mainly because systems commit to an entity without testing it and then trust whatever text arrives. Checks that compare retrieved text against the image, run by vision tools, should catch both wrong-entity errors and misleading-passage errors more reliably than two alternatives: learned rerankers, which score overall similarity, and natural-language chain-of-thought (the model reasoning in prose), which is prone to rationalization. The distinctive prediction is about noise. When we inject passages that are on-topic but wrong, executable checks should degrade far less than chain-of-thought-based defenses. The reason is that an executed check cannot be argued out of its result.

Relation to Prior Work

Four groups of work motivate this design or supply its parts. For visual programs, ViperGPT composes vision tools such as GLIP and BLIP-2 into Python in a single pass and hands knowledge questions to a language model's memory. VDebugger adds a critic and a refiner that read line-level variable traces to fix logic errors. VDebugger reports that about 58% of visual-program errors are logic flaws, but it evaluates only closed-world benchmarks (GQA, TallyQA, RefCOCO) and cannot inspect visual intermediates. We reuse its localize-then-edit structure. The object being debugged, however, is the entity hypothesis and the retrieved facts, which VDebugger does not represent.

For knowledge retrieval, OMGM and QKVQA build strong coarse-to-fine pipelines. OMGM runs entity search, then multimodal reranking, then section extraction. QKVQA adds a question-focused filter and dynamic article selection. Both are feedforward: once a reranker scores an article, nothing downstream tests that choice against the image, and the pipeline never abstains. EnFoRe and EntRAG likewise center retrieval on entities without closing a verification loop.

For noise robustness, VaccineRAG trains multimodal models with a reinforcement learning method (Partial-GRPO) to judge helpful versus harmful references in chain-of-thought. It cuts the accuracy degradation rate under five harmful samples from 42.45% to 10.43% on Qwen2.5-VL. That judgment is still made in prose, which is the channel Thinking in a Crowd shows is vulnerable: Qwen3-8B falls from 96.22% to 77.03% with misleading snippets when thinking is on. This result constrains our design. It argues against letting the same model verbally adjudicate retrieved text, and for delegating the adjudication to executed tools.

The last group constrains how the loop is built. Refinement loops plateau quickly (Yan et al.) and compound errors when feedback is poorly localized (Srivastava et al.). We therefore bound the number of iterations and use explicit rollback to prior states, following Fang et al., rather than open-ended re-prompting. AdaCoder shows that pruning API documentation improves programmatic question answering: GQA accuracy rises from 41.3% to 43.6% at 71.1% fewer prompt tokens. We adopt that pruning so the added verification tools do not bloat the prompt.

Research Program

The system has four parts. First, a candidate generator proposes the top-$k$ entities using an OMGM-style coarse search. Second, for each candidate $e$, a claim extractor reads the retrieved article and selects up to $m$ attributes that satisfy two conditions: they are visually checkable, and they discriminate $e$ from the other candidates. An attribute shared by all candidates is useless for this purpose, so selection is contrastive across the candidate set. Third, a code generator turns each attribute into a call to the vision toolset (detection, counting, color and material queries, text reading). Fourth, a controller ranks candidates by

$$s(e) = \log p_{\text{ret}}(e) + \sum_{j=1}^{m} w_j\, c_j(e),$$

where $p_{\text{ret}}(e)$ is the retriever's probability for candidate $e$, $c_j(e) \in \{-1, 0, +1\}$ records whether check $j$ failed, was inconclusive, or passed, and $w_j$ is a per-check weight. The weight is set by how reliable that check type proves to be on a held-out calibration split. The controller answers from the top candidate only if its checks pass. Otherwise it rolls back, expands $k$ or re-queries once, and abstains after a fixed budget.

The same mechanism covers the reading stage. The answer-bearing sentence in a passage is also checked for consistency with the visual evidence and with the other retrieved chunks. A passage that says "three spans" when the counter finds five is flagged, not rationalized.

We will analyze which check types actually carry signal, how often the vision tools themselves are wrong, and where the loop stops helping. Tool error is the method's own weak point. A detector that hallucinates "red towers" produces a false pass, and we will measure check precision directly against hand-labeled attributes.

Baselines

The naive reference point is a closed-book model: PaLI-X scores 10.8% on INFOSEEK-Human (Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?). We will also run a current open model of the same size as our generator with no retrieval. The simple pipeline reference is CLIP plus the Fusion-in-Decoder reader (FiD), a model that reads many passages jointly, at 18.2% on the same split. The oracle-entity ceiling of 45.6% tells us how much of the entity gap we close.

The strongest existing retrieval method is OMGM, which reports 43.49% on InfoSeek and 50.17% on Encyclopedic-VQA with LLaVA-1.5-7B. QKVQA reports margins of 3.8 and 4.9 points over OMGM on the two benchmarks. The file gives these only as margins over OMGM, not as independent absolute scores. We will rerun both with LLaVA-1.5-7B as the shared generator, on the same Wikipedia knowledge base and the same top-$k$ candidate pool. Our method will use the same generator for claim extraction and code writing, and it will get matched tuning on the same validation split. Beating these baselines means a statistically reliable accuracy gain on clean data at equal generator size, or matched clean accuracy with a clearly smaller degradation under injected noise. We will report latency against QKVQA's 3.98 seconds per question.

For noise robustness, the strongest competitor is VaccineRAG's Partial-GRPO model. We will evaluate it on the same corrupted test sets with the same backbone family (Qwen2.5-VL), and compare accuracy degradation rates at five harmful samples against its reported 10.43%.

For the obvious combination, we build a VDebugger-style critic and refiner with retrieval tools added to its API. It gets the same iteration budget but fixes crashes and logic errors rather than testing entity hypotheses.

Two ablations isolate the key ingredient, executed checks. In the first, the executed checks are replaced by asking the same vision-language model in prose, "does this article match the image?", with rollback and budget kept identical. In the second, the checks are kept but the contrastive attribute selection is removed.

Why not just…?

Why not just use a stronger cross-modal reranker? This is the real threat. OMGM's multimodal reranking already compares image and article, and on clean single-hop questions it may capture most of what our checks do. The difference is in what each method can express. A reranker produces one similarity score and cannot state that a candidate must have two towers when a look-alike has one. It also has no way to decline. The reranker baseline and the contrastive-selection ablation settle this. If OMGM or QKVQA matches us on both clean and corrupted splits, the claim is wrong, and the contribution shrinks to abstention quality alone.

Why not just have the model judge retrieved passages in chain-of-thought, as VaccineRAG does? VaccineRAG's own reduction of degradation shows this helps. But Thinking in a Crowd shows that prose deliberation rationalizes misleading premises, and MATH-Vision finds that chain-of-thought is inconsistent and sometimes harmful in multimodal models. Our prose-verification ablation tests this directly. It uses the same loop and the same budget, and swaps only executed checks for a verbal judgment.

Why not just add retrieval calls to ViperGPT and let VDebugger fix the programs? Such a pipeline repairs programs that crash or compute wrongly. The dominant INFOSEEK failure, however, is a program that runs cleanly on the wrong entity, and a trace-reading critic has no signal to detect it. The augmented-VDebugger baseline measures how much of our gain comes from debugging retrieval hypotheses versus debugging code in general.

Why not just scale the model? The closed-book 10.8% and the 45.6% oracle ceiling both indicate that entity identity, not model size, limits these systems. We will still report a large closed-book model as a reference.

Proposed Experiments

Evaluation uses INFOSEEK (both the Wikidata-generated and human-written splits) and Encyclopedic-VQA as primary benchmarks, with OK-VQA as a check on transfer. ViperGPT reports 51.9% zero-shot on OK-VQA. Metrics are answer accuracy, entity-linking accuracy, selective accuracy under abstention (accuracy on answered questions at matched coverage), and tool calls and latency per question.

We will build corrupted versions of the INFOSEEK and Encyclopedic-VQA test sets, following the helpful, irrelevant, and misleading design of SciAux (SciAux). One corruption type injects passages about visually similar wrong entities. The other edits passages about the correct entity to change a single attribute or the answer fact. Degradation rates are reported at one, three, and five injected passages.

The first decisive check is the entity gap. We compute what fraction of the distance between the 18.2% pipeline and the 45.6% oracle our method recovers, compared with OMGM and QKVQA, using identical candidate pools. The second is the noise curve. On passages about the right entity with one edited attribute, executed checks should degrade clearly less than both VaccineRAG and the prose-verification ablation. If they do not, the central claim fails. The third is attribution. We will label check outcomes against human-annotated attributes to see whether gains come from true rejections or from lucky tool errors, and plot accuracy against the iteration budget to locate the plateau that Yan et al. predicts.