Seeing Twice Before Choosing: Split Perception–Reasoning Verifiers for Multimodal Best-of-N

When a vision-language model is given more inference compute, the usual approach is to sample many candidate solutions and let a separate scoring model, a verifier, pick the best one. On diagram and chart problems this procedure has a specific way of failing. Models often misread the picture: they get a bar height wrong, or invent an angle. Then they reason fluently from the misreading. A verifier that gives each step one overall score tends to reward that fluency. The risk grows with more samples, because a larger pool contains more confident, well-written, wrongly grounded chains for the verifier to prefer. We propose a verifier that grades each step twice. One grade asks whether the visual claim matches the image. The other asks whether the inference follows. Each grade is read off as a probability-weighted expected value over ordinal rubric words rather than as a single argmax verdict. We train and test this verifier on procedurally generated charts and geometric figures, where the generator knows the true value of every visual quantity. That knowledge gives automatic step-level perception labels and test items that cannot have been memorized.

The opportunity matters because current reports of multimodal test-time scaling rest on static benchmarks. On those benchmarks a verifier can recognize a memorized solution, and a policy can use a text shortcut instead of reading the figure. We do not know how much of the reported Best-of-N gain survives on fresh images whose numbers have been redrawn. We also do not know whether the remaining failures come from perception or from reasoning. The proposal answers both questions and tests one concrete fix.

Core Claim

Our claim is that on contamination-free, procedurally varied visual reasoning problems, three things hold. First, most Best-of-N selection errors at large N are perception errors that a single-score verifier cannot separate from reasoning quality. Second, a verifier that scores perception and reasoning in separate channels, each as an expected value over ordinal levels, keeps improving with N in cases where single-score verifiers plateau or decline. Third, the drop in gain between a static benchmark item and its regenerated twin is smaller for the split verifier, which would show that part of the static gains of existing verifiers rides on memorization.

The new ingredient is the split perception channel, trained with labels that come for free from a generator. Probability weighting is the way each channel's verdict is turned into a ranking signal fine enough to rank 128 candidates. We test the two ingredients separately, because either could turn out to be the one doing the work.

Grounding in Prior Work

Several papers motivate the proposal by finding the gap. MATH-Vision built a 3,040-problem competition benchmark. GPT-4o reaches 30.39% on it against 68.82% for humans. Error analysis of GPT-4V there found 31.9% of errors were visual recognition errors and 42.2% were reasoning errors. Perception and reasoning are therefore both large and distinct failure sources, and a verifier should not merge them. Plot Twist probed elementary chart reading with Chart101. Models used title text as a shortcut: accuracy was as high as 79% when the title contained the answer and fell to the 30% range when it did not. Adding grid lines dropped PaLI-3 from 29.5% to 21.3%. More Thinking, Less Seeing? showed that as multimodal reasoning chains lengthen, the model's attention drifts from image tokens toward its own text and hallucination increases. That paper diagnoses the problem but proposes no selection mechanism. We supply one, aimed directly at drifted chains.

Two papers constrain the design. Inverse Scaling in Test-Time Compute and Towards Thinking-Optimal Scaling show that more reasoning can lower accuracy on text tasks. So we cannot assume that larger N or longer chains help, and we must include a length-based baseline. Scaling Test-Time Compute Without Verification or RL is Suboptimal argues that search is only as good as its verifier. This is why the work centers on the verifier and not on the policy model.

The closest existing systems are two multimodal process reward models, which score each intermediate step of a solution. DiscPRM trains a Qwen2.5-VL-7B critic on VTReward-100K, whose step labels come from Monte Carlo rollouts. It outputs one scalar per step and selects among up to 128 candidates. GM-PRM writes free-text critiques along three dimensions (step intent, image alignment, reasoning logic) and corrects the first wrong step. GM-PRM already names image alignment as a separate dimension. We differ in two ways. Our alignment judgment is a calibrated number trained on generator-verified ground truth, not a free-text analysis. And we evaluate on regenerated items rather than only static ones.

Two papers supply the scoring method. Q-Align trains multimodal models to output text levels and reads an expected score from the probabilities of those level words. It reports this as roughly 10% more effective than direct regression, though only for perceptual image quality. TrustJudge shows that expected-value scoring cuts a text judge's pairwise transitivity inconsistency from 15.22% to 4.40%. We carry the Q-Align recipe from image quality to step verification.

The benchmark side borrows its structure from DyVal. DyVal generates reasoning problems as graphs of dependent steps with tunable depth D1 to D4, and found that models strong on static benchmarks scored near zero on its fresh items. We render such graphs as images, which DyVal did not do.

Research Program

The generator. We first build a generator for two families of problems. The first family is charts: bar, line, scatter and pie plots, following Chart101's synthetic plots. The second is plane geometry figures rendered from explicit coordinates. Each problem is a dependency graph of depth D1 to D4. The leaves are visual readings, such as a bar value, a segment length or an intersection point. The internal nodes are arithmetic or geometric inferences. Because the generator holds every leaf value, any perception claim a model makes in a step can be checked automatically against ground truth, using Chart101's 5% tolerance for continuous readings.

We also add controlled nuisance features. These include informative, misleading or absent titles, grid lines, and reordered legends. They let us measure shortcut use directly.

Static twins. For the subset of MathVista, MATH-Vision and DynaMath items that are parametric charts or coordinate geometry, we also produce twins. A twin is the same problem re-rendered with new values and a new layout.

The verifier. We fine-tune Qwen2.5-VL-7B, the same backbone DiscPRM uses, to answer two rubric questions per step. The perception question has four levels: exact, close, off, unsupported. Its labels come from the generator. The reasoning question has three levels: valid, slip, invalid. Its labels come from Monte Carlo rollouts, as in VTReward-100K, conditioned on correct readings.

At inference time, each channel's score is the expected value of its level values under the softmax over the level words. Let $p_t$ and $r_t$ be the perception and reasoning expected values at step $t$, each scaled to $[0,1]$. A candidate trajectory's score is

$$S = \min_t \big(\lambda\, p_t + (1-\lambda)\, r_t\big),$$

with $\lambda$ tuned on held-out generated data. The minimum encodes the idea that one bad step ruins a chain.

Policy models. We use Qwen2.5-VL-7B, InternVL3-8B and MiniCPM-V2.6-8B, the policy models used in the closest prior work. They sample N between 1 and 128 candidates.

Baselines

The naive reference point is the policy model decoding greedily (N = 1), together with majority voting over final answers at each N. Majority voting uses the same samples as every verifier and costs no verifier compute. Any verifier must beat it at matched N to justify itself. We also report an oracle that picks a correct candidate whenever one exists. The gap between majority voting and the oracle shows how much room selection has.

The strongest existing method is a DiscPRM-style scalar step verifier. Its reported results are as follows. DiscPRM reports an F1 of 65.9 on VABench, against 59.9 for VisualPRM. It reports gains of over 5% for Deepeyes and Thyme on MathVista, MME-RealWorld and BLINK, and up to 7.2 points over other multimodal PRMs on MME-RealWorld as N grows to 128. Its result on MATH-Vision, and on any procedurally generated multimodal benchmark, is not available: no reported result found on Lacuna for either. We will run the released checkpoint if one exists. We will also retrain the same architecture on the same mix of VTReward-100K and our generated data, with the same training steps and hyperparameter search budget as our verifier. Its scalar step probability will be aggregated with the same minimum-over-steps rule.

For the generative alternative we use GM-PRM with Refined Best-of-N. It reports gains of +5.9 points on average for MiniCPM-V2.6-8B and +5.6 for InternVL3-8B across five benchmarks including MATH-Vision. We run it at matched total generated tokens, not matched N, since its corrections consume extra compute.

Ablations. Four ablations isolate our ingredients. The first is a single fused rubric: one ordinal question, still read as an expected value, trained on the combined labels. It isolates the value of the split. The second keeps the split but takes the argmax level instead of the expected value. It isolates probability weighting. The third keeps the split but trains the perception channel on rollout labels instead of generator labels. It isolates the value of exact perception supervision. The fourth replaces the learned perception channel with independent re-reading: the base model is asked each extracted elementary question on its own, with no reasoning context, and agreement with the step's claim is scored.

Success criterion. Beating the strongest baseline means higher selected-answer accuracy than the retrained scalar verifier at N = 64 and N = 128 on D3–D4 generated items, across all three policy models, with paired bootstrap confidence intervals that exclude zero. It also means the split verifier's accuracy is still increasing in N where the scalar verifier's has flattened or declined.

Why not just…?

Why not just use the scalar PRM's probability of "correct", which is already continuous? This is the most serious threat. The quantization argument from TrustJudge applies to judges that write out integers, not to classifiers that already output a sigmoid. Our expected-value-versus-argmax ablation and the fused-rubric ablation settle the question directly. If the retrained scalar verifier matches the split verifier, the result is clear: generator-provided perception labels are what matter, and the output format is irrelevant. We would report that as the finding. The bet we are making is narrower. We expect the separate perception channel, not the probability weighting by itself, to pay off. We expect it most on misleading-title and grid-line variants, where the fused score is predicted to stay high for fluent hallucinated chains.

Why not just take a majority vote over the samples and skip the verifier? Majority voting is robust when errors are scattered. Perception errors are not scattered. If a misleading title pushes most samples toward the same wrong reading, the vote amplifies that reading. Plot Twist's title-shortcut results make this likely, but no cited paper measures it under sampling. Our nuisance-controlled items measure it. The diagnostic is whether majority voting's accuracy on misleading-title items falls as N grows.

Why not just prefer shorter chains, since longer thinking causes drift? Thinking-Optimal Scaling and the attention-drift findings make a length penalty a real competitor. We include shortest-majority selection and a length-penalized scalar verifier as baselines. On D4 items, correct chains must be long, so a length prior should hurt there while helping at D1. The split verifier wins only if it beats length heuristics at every depth.

Why not just re-ask the model each visual question in isolation? This independent-re-reading ablation may capture much of the benefit at zero training cost. That would still be a useful result, because it would show the key move is checking perception separately from reasoning. We report it as such. The learned perception channel earns its place only if it beats re-reading. That would happen if, for example, the model reads charts as badly in isolation as in context, as Chart101's elementary-question results suggest it might.

Proposed Experiments

The first experiment draws accuracy-versus-N curves for N from 1 to 128 on generated D1–D4 charts and geometry, for every verifier and policy pair. Each selection error at N = 128 is then labeled automatically as perception or reasoning, using the generator's ground truth. The decisive check is twofold. Is the majority of scalar-verifier errors at large N perception errors? And does the split verifier reduce that category specifically?

The second experiment is the twin study. For each verifier we compute its Best-of-N gain over majority voting on the original MathVista, MATH-Vision and DynaMath items, and again on their regenerated twins. The difference between the two gains measures how much of the static gain depends on memorized items. We predict this difference is larger for the scalar and generative baselines than for the split verifier.

Third, we run selection on RH-Bench across reasoning budgets. We test whether split-verifier selection raises the perception-task accuracy of chosen answers relative to scalar selection. That would be direct evidence that it filters out drifted chains. Ocean-R1-7B's reported RH-AUC of 0.63 serves as a reference point for grounding without selection.

Finally, we check calibration. On generated data, the perception channel's expected value should rank steps by true reading error. We measure this with rank correlation against the absolute error of each reading. If that correlation is weak, the channel is not doing what we claim, whatever the downstream accuracy shows.