When a vision-language model solves a chart or geometry problem and gets the final answer wrong, reinforcement learning usually blames every token of its reasoning equally. Benchmarks show that this blame is badly misplaced. Models often read a chart's values incorrectly while still reasoning soundly about them. They also often read every value correctly and then botch the arithmetic. In some cases they skip the image entirely and answer from the chart title. This note proposes a training method that finds, from the model's own sampled continuations, the steps where the outcome is actually decided. It then concentrates credit on those steps, so that the effective credit window is short at decisive steps and long elsewhere. It also uses a three-level task taxonomy (perceive, relate, compute) to test which kind of step turns out to be decisive.
The idea in two sentences is this. At a handful of points in a reasoning trace, we resample several continuations from the same prefix. How much their final rewards disagree tells us how "critical" that point is. Credit is then assigned in segments that end at critical points, with each segment judged by how much it moved the estimated success rate. This turns a decades-old reinforcement learning idea, varying the bootstrapping horizon by state criticality, into something usable for multimodal language models. That idea previously required a human to supply the criticality function. The taxonomy is used to ask an open empirical question: is perception or symbolic deduction the real bottleneck during learning?
Core Claim
The Concept of Criticality in Reinforcement Learning and its algorithmic companion Criticality-Based Varying Step-Number Algorithm define a state's criticality as the spread of action values across choices at that state. Their algorithm, Criticality-based Varying Stepsize (CVS), adapts the n-step bootstrapping horizon to it. Here an "n-step horizon" means how many future steps of reward are summed before falling back on a value estimate. In uncritical stretches the horizon grows toward a full Monte Carlo return; at critical junctions it shrinks toward a one-step update. CVS converged in about 100 episodes on Shooter where Q-learning needed 1,400. However, it was tested only on toy games, with criticality handed over by a hand-crafted oracle.
Our claim is that sibling rollouts in language-model reinforcement learning already give an unbiased Monte Carlo estimate of exactly this quantity. Sibling rollouts are continuations sampled from a shared prefix, as in Multi-Branch Policy Optimization (MBPO). The open problem CVS left, how to obtain criticality without an oracle, is therefore solvable in this setting. Placing branch points and credit boundaries by estimated criticality should beat MBPO's fixed trigger at matched compute. MBPO branches wherever the model emits a <look> token.
The gain should be largest on elementary perception probes. Outcome-only reinforcement learning currently lets models shortcut those probes, which produces the "harder is easier" paradox documented in Plot Twist. There, Gemini Pro Vision was about 10% less accurate on elementary chart reading than on complex ChartQA questions about the same images.
Research Program
The base model is a Qwen2.5-VL-7B-class model. It is given a brief cold start so that it writes its reasoning as short steps, each tagged by tier. Perceive steps read values, labels and elements from the image. Relate steps state spatial, ordinal or structural relations. Compute steps do arithmetic or deduction. The tags serve analysis and placement; the rewards never depend on tag correctness. Training uses only a verifiable final-answer reward.
For each prompt, we sample a group of main rollouts. At a budgeted number $M$ of step boundaries per rollout, we draw $K$ short continuations to completion and score them. For a binary reward with success rate $\hat p$ across those $K$ continuations, the criticality estimate is
$$\hat h(s) = 4\,\hat p\,(1-\hat p).$$
This is the reward variance rescaled to lie in $[0,1]$. It follows CVS's definition, criticality as outcome variance over choices, but replaces the oracle with sampled continuations. Here $\hat p$ also serves as a critic-free value estimate $\hat V(s)$ of the prefix. A state is uncritical ($\hat h \approx 0$) when the outcome is already settled. It is maximally critical when what the model writes next decides the outcome.
Following CVS, we accumulate $\hat h$ along the trace. A segment boundary is placed when the running sum crosses a threshold $\tau$. Every token in segment $j$, running from boundary $b_j$ to $b_{j+1}$, receives the advantage
$$A_t = \hat V(s_{b_{j+1}}) - \hat V(s_{b_j}),$$
where $\hat V$ at the trace's end is the realized reward. Long uncritical stretches therefore share one Monte Carlo-style credit signal. Steps adjacent to a critical point receive their own sharply localized signal.
To spend the branching budget well, boundary candidates are chosen adaptively in two stages. A cheap first pass ranks step boundaries by the policy's token-level entropy. Continuations are then spent only on the top-ranked ones. We also report the entropy-only variant, because if it works as well, the expensive estimate is unnecessary. The policy update is a standard clipped group-relative objective that uses these per-segment advantages. Following the instability noted for GRPO-λ, we clip negative advantages and monitor divergence from the reference model.
Training data is MBPO's MMRL-18K mix plus synthetic charts. The synthetic charts follow the programmatic generation of Synthesize Step-by-Step, with ground-truth values known, along with scientific figures from VeriSciQA. The ground-truth chart values are used only for evaluation and for one oracle baseline, never by the proposed method.
The analytical half of the program resolves a tension the literature leaves open. MATH-Vision attributes 31.9% of GPT-4V's errors to vision recognition and 42.2% to reasoning. Tier-structured studies like MVU-Eval and Around the World in 24 Hours show performance cliffs between tiers. None of this says which tier is critical in the reinforcement learning sense, meaning where outcomes are actually decided during sampling. We will log $\hat h$ by tier across training, by benchmark and by difficulty level. The prediction is that criticality concentrates at perceive steps on charts and at compute steps on competition geometry. A single hand-set tier rule would then be wrong somewhere, which is why criticality is estimated rather than declared.
Baselines
The naive reference is standard group-relative policy optimization with a single trajectory-level advantage. It is trained on the same data, from the same checkpoint, with the same number of sampled tokens. Any branching method gets extra samples, so the naive baseline is given a larger rollout group until total generated tokens match.
The strongest existing method is MBPO (Multi-Branch Policy Optimization). It branches by breadth-first search at <look> tokens and contrasts sibling branches. It reports 52.6% on MathVerse and 87.7% on ChartQA with a 7B Qwen-VL model, and up to +8.7 points over GRPO on K12 when matched for TFLOPs. We will rerun MBPO from our checkpoint on MMRL-18K under the same TFLOP budget and the same hyperparameter search size. We will also quote its published numbers. Beating it means at least matching it on MathVerse and ChartQA at equal compute, while clearly exceeding it on elementary chart probes. On those probes, a gain of more than three points in collective accuracy, outside seed variance across three seeds, is the bar. MBPO reports no MathVision number, and there is no reported result on Lacuna for any credit-assignment method on Chart101.
We also run GRPO-λ (GRPO-λ) ported to the multimodal model, with its "both" weighting. It reported a 10-point AIME24 gain in text-only math. It is the strongest flat-decay, critic-free alternative, and we tune λ with the same budget.
We include a process-reward baseline in the style of CAPO, which penalizes judge-flagged steps. Its judge is a large multimodal model; CAPO reported 3.5% on text math with a 72B judge. Its judge compute is counted against the budget.
Three ablations isolate the key ingredient. The first uses the same segment advantage but places boundaries uniformly every $k$ steps, which tests whether criticality-driven placement matters. The second fixes the horizon by tier: short at perceive steps, long elsewhere, plus the reverse rule. This tests whether a declared taxonomy can replace measured criticality. The third ranks boundaries by entropy only, with no continuations.
Why not just…?
Why not just give dense rewards for correct perception steps? On synthetic charts we know the true values, so rewarding correctly read numbers is cheap and likely effective there. This is a real threat on chart data, and we run it as an oracle-supervision baseline. Its limitation is coverage. Competition geometry in MATH-Vision and the scenario reasoning in GeoSQA have no perceptual ground truth. In GeoSQA, even perfect text descriptions of diagrams left models near chance, so the failure there is not perceptual at all. The test is whether our label-free method closes most of the gap to the oracle on charts while also improving geometry, where the oracle cannot be applied.
Why not just use MBPO with more branches? MBPO's trigger fires only where the model chooses to look again. The harder-is-easier evidence suggests models skip looking when a textual shortcut is available, so the branch points would be absent exactly where shortcuts occur. The compute-matched comparison, together with the title-ablation test below, settles this. If MBPO with a larger budget matches us, criticality placement adds nothing, and we will report that.
Why not just use a process reward model to judge each step? Judges see the same image the policy misreads, and DCoT shows how downstream reasoning collapses when first-pass visual grounding fails. A multimodal judge inherits the 31.9% vision-error mode. Our signal comes from outcome-verified continuations and needs no judge. The CAPO-style baseline measures whether judge noise matters in practice.
Why not just label steps by tier and fix horizons accordingly? Because which tier is critical is the open question. The tier-rule ablations, run in both directions, will show whether either fixed assignment helps everywhere. Our prediction is that each helps on one benchmark family and hurts on the other.
Proposed Experiments and Decisive Checks
Evaluation covers MathVerse, ChartQA, MATH-Vision (reference points: GPT-4o at 30.39%, human at 68.82%, random at about 5.86%), MathVista, Chart101 elementary questions with collective accuracy, and K12. Plot Twist reports collective accuracy below 25% for all tested models on Chart101. We also build a title-ablated variant of ChartQA. Plot Twist found accuracy as high as 79% when the title contained the answer and in the 30% range when the title was removed or misleading.
Four checks decide the proposal. The first is validity: $\hat h$ should peak at the first erroneous step in failed traces. We verify this against human-located first errors on 300 MATH-Vision and chart traces, in the spirit of the first-error localization in InT. The method is in trouble if this correlation is near zero. The second is compute-matched performance against MBPO, as specified above. The third is shortcut reduction: the drop from titled to title-ablated charts should shrink relative to GRPO and MBPO, and Chart101 elementary accuracy should rise more than ChartQA accuracy. The fourth is the tier analysis: the measured distribution of criticality across perceive, relate and compute steps, reported per benchmark family. This analysis is useful even if the method's accuracy gains are modest.
References
- The Concept of Criticality in Reinforcement Learning
- Criticality-based Varying Step-number Algorithm for Reinforcement Learning
- Multi-Branch Policy Optimization for Multimodal Large Language Models
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- GRPO-$\lambda$: Credit Assignment improves LLM Reasoning
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs
- Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
- CAPO: Towards Enhancing LLM Reasoning through Verifiable Generative Credit Assignment
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- InT: Self-Proposed Interventions Enable Credit Assignment in LLM Reasoning