When a vision-language model solves a chart or geometry problem step by step, it usually never says what it looked at. It reads the image once, then reasons in text. The evidence that this goes wrong is direct. Models answer hard chart questions better than easy ones because they guess from chart titles rather than reading the plotted values. On competition math with diagrams, roughly a third of GPT-4V's errors are misreadings of the picture, and a larger share are reasoning failures built on top of it. We propose a different inference procedure. Every visual fact the model relies on must first be written down as a short, numbered textual claim, a "premise" such as "#3: the bar for 2019 reaches about 42 on the y-axis." Each premise is produced by a targeted description of one image region. It is then checked independently against the image before the model may use it. Every later reasoning step must cite the premises it depends on.
This turns the model's hidden perception into an explicit record that can be inspected. That has three payoffs. Perception errors can be caught and repaired locally instead of by resampling the whole answer. Reasoning steps can be verified in isolation, as has been done for text-only math. And we can directly test whether the final answer actually depends on what the model claims to have seen, which is the "Clever Hans" question that current chain-of-thought methods cannot answer. The idea combines three threads. Caption synthesis supplies the mechanism: text written by perception tools serves as the intermediate representation. Chain-of-thought prompting, where the model writes out intermediate reasoning before answering, supplies the structure. The multimodal reasoning gap supplies the failure we target and the benchmarks we test on.
Core Claim
We claim that discretizing perception into cited, independently verified textual premises during chain-of-thought improves accuracy on perception-heavy reasoning benchmarks over pixel-level "look again" methods given the same compute. More distinctively, we claim it makes answers counterfactually dependent on visual evidence: if a verified premise is changed, the answer changes accordingly. If the method raises accuracy but answers do not track their premises, the premises are decoration, and we will report that as a negative result.
What Exists and Where This Sits
Several pieces of evidence motivate the work. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset built a 3,040-problem competition benchmark and attributed 31.9% of GPT-4V's errors to vision recognition and 42.2% to reasoning. It also found that plain chain-of-thought sometimes hurts multimodal models. Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details probed elementary chart reading. It found accuracy as high as 79% when the title contains the answer and in the 30% range when the title is misleading. Collective accuracy, meaning every elementary question on a chart answered correctly, often falls below 25%. Together these say the model's reasoning is not anchored to what it perceives. They also set a ceiling for us: perfect perception would remove at most about a third of GPT-4V's MATH-Vision errors. The remaining errors must be addressed by verifying the reasoning steps themselves.
The method borrows from two places. Deductive Verification of Chain-of-Thought Reasoning introduced the "Natural Program" format for text math. In that format, premises are numbered, each step cites the premises it uses, and steps are verified one at a time by voting. It reports that whole-chain verification by GPT-3.5-turbo is near 52%, close to coin-flipping, while step-wise checking does substantially better. We extend the premise list to include visual premises that come from the image rather than from the question text. That paper never faced this problem, because its premises were given. Caption-fusion work such as FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions shows that combining detector, OCR and captioner outputs through a language model recovers small objects and text that generic captions omit. FuseCap does this offline and without regard to the task. We do it on demand, one region at a time, for the specific fact a reasoning step needs.
The closest prior system is Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning (VCTP). Its loop works as follows: a language model picks objects relevant to the question, requests regional captions, proposes an answer, and confirms its free-form rationale with CLIP similarity, a global image-text matching score. It improved A-OKVQA from 41.5% to 46.4%. We differ in three ways. Our verified unit is an atomic premise, not a whole rationale. Our check is a targeted closed-form re-query, not CLIP, which is known to be weak on numbers, positions and relations. And we evaluate on math and charts, where the gap is largest, rather than commonsense VQA.
Three methods constrain the design and also serve as competitors. Simple o3: Towards Interleaved Vision-Language Reasoning lets the model crop and zoom during reasoning, then reason over the cropped pixels. TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding and Interleaved-Modal Chain-of-Thought re-inject selected visual feature vectors mid-reasoning. All three re-access the image, but what they re-access is continuous and cannot be checked by anything outside the model. Our bet is the opposite one. Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA and AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive Reasoning both support it. The first raised ChartQA human-split accuracy from 38.2% to 53.9% by going through tables and executable rationales. The second reached 81.6% on Geometry3K and 99% judged logical soundness through a geometry-specific formal language. We aim for the discreteness of these methods without a domain grammar.
Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (PEARL) gates reinforcement-learning updates on perception checklists derived from ground-truth annotations. Our premises are produced and checked at inference time without annotations, so the two approaches are complementary rather than redundant. Finally, GeoSQA is a warning. Even human-written transcriptions of geography diagrams left text models near chance. Converting images to text does not by itself create reasoning ability, which is why step verification is part of the method and not an optional extra.
Research Program
The procedure runs on one open vision-language backbone, Qwen2.5-VL-7B (used by both PEARL and Simple o3), with no training in the main study. The model reasons in a fixed format with three kinds of lines. A look line names a region (a bounding box) and a question about it. Rather than reasoning over the crop's pixels, the system answers the look line by producing a short targeted description of that region. The description comes from the backbone itself, prompted with only the crop. Where applicable, it is fused with outputs from specialist tools: OCR for text and tick labels, a chart-to-table converter for plotted values, and a detector for object counts. The resulting description is written as a numbered visual premise. Step lines perform deductions and must cite premise numbers. An answer line closes the chain.
Each visual premise is checked before it enters the context. The checker rephrases the premise into a closed-form question, for example "Is the 2019 bar's value between 40 and 44?" It asks this question in a fresh context under several conditions: with the original crop, with a slightly shifted crop, and with a cosmetically perturbed rendering where possible. The perturbations reuse the Plot Twist manipulations, such as adding grid lines or masking the title. Where a specialist tool covers the premise type, its output must also agree. A premise is accepted only if all independent reads agree. Otherwise the system reports the disagreement back to the reasoning chain, which must re-look or mark the fact as uncertain. Step lines are verified in isolation, Natural-Program style: the verifier sees only the cited premises and the step, and votes on validity. A rejected step is regenerated from that point rather than restarting the chain.
This design produces something no current method gives: an automatic per-problem split of failures into "wrong premise accepted," "right premises, invalid step," and "valid chain, answer not supported." We will report this split against MATH-Vision's human error taxonomy to see whether the automatic attribution agrees with manual labeling.
Baselines
The naive reference point is the same backbone answering directly with no reasoning, alongside the same backbone with ordinary step-by-step prompting. Both use the same prompts, decoding settings and answer extraction as our method. We also quote published context. On MathVista, the benchmark reports GPT-4V at 49.9% and humans at 60.3% (MathVista). On MATH-Vision, GPT-4o scores 30.39% and the best open model 16.97%.
The strongest compute-matched competitor is self-consistency over ordinary chain-of-thought: sample many chains and take the majority answer. Our method spends extra forward passes on re-looks and checks. A reviewer can fairly ask whether the same budget spent on more ordinary samples would do as well. We will match total generated tokens plus image-encoder passes per problem and sweep the sample count until budgets equal ours. No reported result on Lacuna was found for self-consistency on these benchmarks with this backbone, so we will establish the number ourselves.
The strongest published methods are TVI-CoT and Simple o3. TVI-CoT reports 76.6% on MathVista and 60.2% on MathVerse, but with Qwen3-VL-8B. MM-Verify reports 65.3% on MathVista with a 7B reasoner, a 7B verifier and 12 rollouts, against GPT-4o's 63.8% (MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification). These published numbers use different backbones and some include training, so they serve as context, not as head-to-head targets.
The fair head-to-head is a training-free reimplementation of the Simple o3 crop-and-zoom loop on our backbone. It uses identical region proposals and identical budget, and differs from our method only in what follows a look: the crop's pixels are fed back into the reasoning context instead of a verified textual premise. For charts we also compare to MatCha with LAMENDA, which reports 72.6% relaxed accuracy on ChartQA. For geometry we compare to AutoGPS at 81.6% on Geometry3K, which we treat as a domain-specific ceiling rather than a target.
Three ablations isolate the key ingredients. In the first, looks produce textual premises but nothing is checked. In the second, premises are checked but steps are not verified. In the third, premises are produced but citation is not required. Beating the Simple o3-style pixel loop and matched self-consistency on MATH-Vision and MathVista, with confidence intervals excluding zero, counts as a win. If premise checking adds nothing over unverified premises, our central mechanism is wrong.
Why not just…?
Why not just run self-consistency with a strong model's ordinary chain-of-thought? This is the most serious threat, because extra samples are cheap and often help. However, majority voting cannot fix a misreading that every sample shares. Plot Twist shows that title-driven guessing is systematic, not random: accuracy falls into the 30% range under misleading titles. Self-consistency should therefore fail most on exactly the perturbed and misleading-title items. The compute-matched comparison, broken down by Plot Twist perturbation condition, will settle this. If self-consistency matches us there too, we will concede that the gain comes from compute, not structure.
Why not just caption the whole image once and reason in text? One static description cannot anticipate which tick value or angle step five will need. FuseCap's motivation is precisely that generic captions drop small text and numbers, and VCTP found iterative regional captioning beat static prompting on A-OKVQA. GeoSQA adds that even careful human transcription did not rescue reasoning. We include a "dense caption once, then text chain-of-thought" baseline at matched budget, where the budget goes into a longer and richer caption. This tests whether targeting matters, or whether a better upfront description is enough.
Why not just crop and zoom on pixels, as Simple o3 does? Cropping clearly helps perception: Simple o3 reports +12.9% on V*. But the cropped pixels go back into the same opaque process that makes the original errors, and nothing outside the model can check them. The head-to-head ablation above changes only this one factor. If pixel re-looks equal our accuracy, the counterfactual test below still distinguishes the two methods. Only explicit premises allow us to intervene on what the model "saw" and measure whether the answer follows.
Why not just train with perception-aware rewards, as PEARL does? PEARL's +9.7% on MathVerse is strong, but it relies on checklists derived from ground-truth annotations available only during training. Our method needs no annotations and runs on any image at test time. The approaches are complementary. If budget allows, we will apply our procedure on top of a PEARL-style checkpoint to check whether the gains stack.
Proposed Experiments
The main evaluation covers four benchmarks. MATH-Vision, which reports its own error taxonomy, lets us compare automatic and manual attribution. MathVista and MathVerse provide breadth. ChartQA plus the Plot Twist elementary probes cover charts, with grid-line, marker-style and misleading-title perturbations applied to probe robustness. Geometry3K serves as a domain where a formal system sets an upper reference. We report accuracy with bootstrap intervals, collective per-chart accuracy, and the accuracy drop under perturbation. We also report the precision of premise checks, measured on a hand-labeled sample of 300 premises spanning charts, geometry and natural images.
The decisive check is counterfactual premise editing. For problems the method answers correctly, we change one cited, verified premise to a different plausible value and let the chain continue from that point. A faithful system's answer should change exactly as the edited value implies. We also run the reverse: edit an uncited premise, after which the answer should stay fixed. We will compare the rate of answer change against ordinary chain-of-thought, whose stated visual claims we edit in the same way. A large gap between the two methods would show that cited premises make answers depend on evidence. No gap would mean the format is cosmetic, regardless of accuracy.
A second check targets where the gains come from. If gains concentrate in items MATH-Vision labels as vision-recognition errors, and the step verifier separately reduces reasoning-error items, the decomposition is working as designed. If gains appear only on items answerable from the question text alone, the method is exploiting text shortcuts like everything before it.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Deductive Verification of Chain-of-Thought Reasoning
- FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
- Visual Chain-of-Thought Prompting for Knowledge-Based Visual Reasoning
- Simple o3: Towards Interleaved Vision-Language Reasoning
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding
- Interleaved-Modal Chain-of-Thought
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive Reasoning
- Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification