Vision-language models fail on diagram math and chart questions for two different reasons. Sometimes they misread the picture. Other times they read it correctly and then reason badly. On MATH-Vision, GPT-4V's errors split into 31.9% visual recognition failures and 42.2% reasoning failures with correct recognition (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). A popular test-time fix is overgenerate-and-rank: sample many full answers and let a second model, the "verifier" or "judge," pick the best one. This fix has a specific weakness in the visual setting. A multimodal judge that reads a fluent solution tends to accept it even when the solution misstates what the image shows (Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation). The persuasive reasoning hides the perceptual hallucination.
We propose to split candidate generation and selection into two stages at test time. First, the model writes several short, structured percepts: lists of atomic visual claims such as "bar for 2019 ≈ 42" or "angle ABC is marked 30°." A claim checker looks only at the image and one claim at a time, and it never sees any downstream reasoning. Second, a text-only reasoner, which does not see the image, writes several solution chains from the surviving percepts, and a separate selector ranks those chains. The final answer combines the two scores. The central claim is that hiding the reasoning from the perception judge removes the anchoring that corrupts holistic multimodal judges. We further claim that this factorization beats flat best-of-N at an equal token budget, and that the gain shows up mainly on the perception-error category.
Core Claim and Position in Prior Work
Three lines of prior work each supply part of this design, and none of them combines the parts. The first line does training-time separation of perception and reasoning. MoCA trains a Qwen2.5-VL-7B policy with reinforcement learning to emit <recognition> facts before <think> steps. It rewards the recognition text when a blindfolded text-only proxy can solve the problem from that text alone (Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning). Video-DPL does the same for video, using timestamped evidence units and a consistency reward from a reference model denied video access (Alternating Perception-Reasoning for Hallucination-Resistant Video Understanding). Both decode a single path at inference. We reuse their output format and their blindfolded-reasoner idea, but we move the separation into test-time selection over multiple candidates.
The second line does test-time selection over monolithic outputs. MM-Verify samples 12 full multimodal rollouts and scores each end to end with a trained 7B verifier (MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification). RASO generates short answer candidates from captions and picks one with a lightweight selector (Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge). Neither can say whether a rejected candidate failed at seeing or at thinking. MM-Verify's verifier also reads the full fluent chain, which is exactly the setting where response anchoring occurs.
The third line motivates the design and constrains it. Park et al. show that judges favor polished reasoning over visual evidence. They fix this by training a better holistic judge, Perception-Judge-Flex-7B, which raises perceptual-error identification from 69.5% to 85.7% on their perturbed-judgment dataset (PPJD). We take their finding as evidence that the cause is the judge's input, and we test a structural remedy: never show the judge the reasoning at all. FlowScore supplies the reasoning-side method (Hallucination Detection from Structural Reasoning Model). It represents a solution as a graph of conditions and steps, with each step citing its parents, checks each node only against those parents, and reports 76.59% average precision on MATH. That work is text-only and assumes the premises are given. In our pipeline, the verified visual claims become those premises.
The selection rule is a weighted vote over answers. Let $I$ be the image, $x_1,\dots,x_K$ the sampled percepts, and $c_{k1},\dots,c_{kM}$ the reasoning chains generated from percept $x_k$. Define $s(x_k\mid I)$ as the percept score: the product of the claim checker's "yes" probabilities over the claims in $x_k$. Define $r(c)$ as the reasoning selector's score for a chain. Then the chosen answer is
$$\hat a=\arg\max_a \sum_{k=1}^{K} s(x_k\mid I)\,\frac{1}{M}\sum_{m=1}^{M} r(c_{km})\,\mathbb{1}[\mathrm{ans}(c_{km})=a].$$
Two properties follow from this form. Because percepts are short and are scored before any reasoning is sampled, low-scoring percepts can be dropped early. Reasoning is then spent only on the top $j$ percepts, so cost grows roughly as $K$ short samples plus $jM$ long ones, instead of $KM$ long ones. And because each chain's evidence is attributed to one percept, every failure can be traced to either the perception stage or the reasoning stage.
Research Program
Claim checker. We fine-tune Qwen2.5-VL-7B to output a single yes/no token for each image–claim pair. The loss is supervised fine-tuning rather than contrastive learning, because GMR found that SFT's weighting of hard examples beats contrastive losses for multimodal rerankers and that binary tokens match multi-token scoring (Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking).
Training data for the checker. Positive and negative claims come from three sources. We render synthetic charts and diagrams whose underlying values are known, in the style of Chart101's synthetic plots. We use VeriSciQA's cross-modally verified figures (VeriSciQA). We also perturb counts, colors, and spatial relations as PPJD does, but at the level of single claims rather than whole responses. Plot Twist found that models read answers off chart titles instead of the plotted data (Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details). To target that shortcut, training images include misleading titles and added gridlines, and the checker must answer from the plotted marks.
Percept and reasoning generation. Percepts are sampled from the base vision-language model with MoCA-style recognition tags. Reasoning chains come from a text-only model (Qwen2.5-14B, matching MoCA's proxy) prompted to write FlowScore-style graphs whose roots are percept claims. The reasoning selector $r$ is FlowScore's local-parent verification. A simpler variant sets $r\equiv 1$, which reduces the second stage to a percept-weighted majority vote. This variant is kept as an ablation.
Scope. Our design inherits MoCA's textual bottleneck: some spatial relations do not reduce to short claims. We therefore report results by MATH-Vision subject. We expect the method to help least on transformation and solid geometry, where GPT-4o already falls to about 18%, and we treat that outcome as a stated limit rather than something to hide.
Baselines
Greedy decoding. The naive reference point is a single greedy chain-of-thought pass from the same Qwen2.5-VL-7B generator. Every other system is reported as a gain over this pass.
Self-consistency. This baseline samples full multimodal chains and takes the majority answer. The number of chains is set so that total generated tokens match our pipeline, with temperature tuned on the same validation split we use. It is the cheapest serious competitor.
MM-Verify. The strongest existing method is MM-Reasoner plus MM-Verifier with 12 rollouts, which reports 65.3 accuracy on MathVista, against 63.8 for GPT-4o in the same paper. We will run it at 12 rollouts and at our matched token budget. Beating it means exceeding 65.3 on MathVista at no more generated tokens, and exceeding it on MATH-Vision, where the only reported reference in our file is GPT-4o at 30.39% with humans at 68.82%.
Debiased holistic judge. This baseline is best-of-N over full chains using Perception-Judge-Flex-7B as the selector. It isolates whether a debiased holistic judge already captures our benefit. We will give it the same candidate pools that our method draws from.
MoCA, single path. We evaluate VL-Scaler-7B decoding once, to test whether training-time separation already captures the benefit without any test-time selection.
Ablations. Four ablations target the key ingredients. In the first, the claim checker sees each claim together with the downstream reasoning chain. This is the anchoring ablation and the central test of the core claim. In the second, $K=1$, so percepts are not branched. In the third, $M=1$, so reasoning is not branched. In the fourth, the reasoner is given the image, which removes the blindfold.
Why not just…?
Why not just use self-consistency? Majority voting amplifies errors that samples share. On HumanEval, FlowScore reports sampling-based detection near 25% AP. On charts, Plot Twist shows models converge on the same title-driven answer, with accuracy falling to the 30% range when titles are removed or misleading. Our matched-budget comparison on Chart101's misleading-title split tests this directly. If voting matches our method there, the percept checker adds nothing.
Why not just best-of-N with Perception-Judge-Flex? This is the real threat. Its 85.7% perceptual-error identification shows that a holistic judge can be substantially debiased. Our position is that removing reasoning from the judge's input should work at least as well, and that it does so at lower cost and with attribution of each failure. Two outcomes settle this. If the anchoring ablation shows no gap on PPJD, and Flex-judge best-of-N matches our accuracy on MathVista and MATH-Vision, then the factorization's contribution reduces to compute savings, and we will report it that way.
Why not just train a better single-path model like MoCA? Training and selection are complementary, but a single path cannot recover from one bad reading of the image. The reported V* score for MoCA is 86.6, but its DUDE score is 45.1 against 52.7 for GPT-4o. This suggests that perception gains from training alone are uneven across tasks. We will run our selection pipeline on top of VL-Scaler-7B percepts to measure whether selection adds accuracy beyond training.
Why not just caption the image and hand it to a strong text model? That setup is our pipeline with $K=1$ and no percept checking. GeoSQA shows that even human-written descriptions leave reasoning failures in place, which is why we also select among reasoning chains. The $K=1$ ablation measures how much of the gain comes from choosing among readings of the image rather than committing to one.
Proposed Experiments and Decisive Checks
We evaluate on MathVista, MATH-Vision, and Chart101 (range accuracy within 5%, plus collective accuracy), and on PPJD for judge behavior. Four checks decide the proposal.
Anchoring removal. The claim checker should flag perceptually perturbed claims more accurately when the reasoning is hidden than when it is shown, measured on PPJD-derived pairs against the 69.5% and 85.7% reference points.
Attribution. We annotate 300 MATH-Vision failures using the paper's error taxonomy. Relative to MM-Verify, the error reduction should concentrate in the vision-recognition category. The 42.2% reasoning share should also shrink as reasoning branching is added.
Budget curves. We plot accuracy against generated tokens for our method, self-consistency, and MM-Verify on MathVista. Our curve should lie above both. We also report percept oracle coverage, meaning the fraction of questions where at least one percept is fully correct. RASO and multimodal reranking work both show that candidate pools cap what any selector can achieve: OK-VQA ensemble coverage is 81.9%, and oracle reranking reaches 64.4% versus 57.2% achieved (Multimodal Reranking for Knowledge-Intensive Visual Question Answering).
Shortcut robustness. On Chart101 with misleading titles, our accuracy drop should be clearly smaller than the baselines' drop.
If the first check fails, meaning hiding the reasoning does not help the checker, the core claim is false regardless of the headline accuracy.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge for Mathematical and Scientific Reasoning Evaluation
- Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning
- Alternating Perception-Reasoning for Hallucination-Resistant Video Understanding
- MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification
- Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge
- Hallucination Detection from Structural Reasoning Model
- Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Multimodal Reranking for Knowledge-Intensive Visual Question Answering