When a vision-language model gets a chart or geometry question wrong, it can be for two very different reasons. It may have misread the picture, or it may have read the picture correctly and then reasoned badly. When it gets a question right, it may have used the picture, or it may have guessed from the question text alone. Current adaptive curricula for reinforcement-learning post-training ignore all of this. These curricula are training schedules that choose which prompts to practice based on how the model is doing right now, and they score each prompt with one number: the fraction of sampled answers that are correct. We propose to replace that single number with three cheap per-prompt measurements. The first is the pass rate with the image. The second is the pass rate when the image is swapped for a faithful text description. The third is the pass rate with no image at all. Each measurement is re-estimated online as the model changes. A scheduler then uses these measurements to choose prompts and to shift training weight between perception practice and reasoning practice.
This matters because the evidence gathered so far says the multimodal reasoning gap is a mix of two failure modes that change over time, and a single pass rate hides both. Error analysis on MATH-Vision attributes 31.9% of GPT-4V errors to misrecognition and 42.2% to reasoning. Plot Twist shows that models look competent on complex chart questions while failing elementary readings of the same charts. Its accuracy reaches 79% when the chart title contains the answer and falls to about 30% when the title is removed. A pass-rate scheduler treats such a title-solvable prompt as a well-learned or "medium" prompt and keeps spending rollouts on it, even though those rollouts teach the model nothing about looking at the image.
Core Claim
Consider a per-prompt difficulty signal with three parts, all measured online during RL post-training. Let $p^{\text{img}}_i$ be prompt $i$'s pass rate with the image. Let $p^{\text{txt}}_i$ be its pass rate when the image is replaced by a description of its perceptual content. Let $p^{\text{blind}}_i$ be its pass rate with the question text only. We claim this signal allocates a fixed rollout budget better than a scalar pass rate does, and that its advantage comes from two sources: it stops rewarding shortcut-solvable prompts, and it detects when the bottleneck moves between perception and reasoning. The sampling score we use is
$$w_i = 4\,p^{\text{img}}_i\,(1-p^{\text{img}}_i)\,\bigl(1-p^{\text{blind}}_i\bigr),$$
where the first factor is the variance of a Bernoulli reward. Group Relative Policy Optimization (GRPO), a policy-gradient method that scores each sampled answer against the other answers drawn for the same prompt, gets a signal whose strength scales with this variance, so the signal vanishes at pass rates of 0 and 1. The second factor removes prompts the model can solve without looking. The perception gap $g_i = p^{\text{txt}}_i - p^{\text{img}}_i$ separates two kinds of failing prompt. Prompts that fail with the image but succeed from a description are perception-bottlenecked. Prompts that fail both ways are reasoning-bottlenecked. The mean gap over frontier prompts, compared with the mean reasoning deficit $1-p^{\text{txt}}$, sets the fraction $\lambda_t$ of each batch drawn from a perception-probe pool rather than the main reasoning pool. The probe pool contains elementary reading questions generated automatically from chart tables or diagram annotations.
The new element is using the with-image versus without-image contrast online, as a routing signal inside the RL loop that changes over time. It is not an offline data filter and not an evaluation diagnostic.
How This Sits Against Prior Work
The motivating evidence comes from the benchmark papers above. GeoSQA replaces diagrams with rich human descriptions and finds that text models still fail, so the description condition genuinely isolates reasoning failures from perception failures. That property is the one our $p^{\text{txt}}$ measurement relies on.
Two staged curricula disagree about the order of training, and that disagreement is the second motivation. From Seeing to Thinking trains perception with RL first, then text-only math, then visual reasoning, and reports that reversing the order hurts. Learning When to Look starts from a text-only reasoning warm-up and then anchors reasoning with inserted perception steps. Both use fixed stage boundaries. We treat their disagreement as evidence that the right emphasis depends on where the model currently fails. The controller $\lambda_t$ lets that emphasis move in either direction. From Seeing to Thinking is also the closest prior work on mechanism. It builds its Stage 1 data by keeping cases where the model fails on the raw image but succeeds with a caption. That is our $g_i$, but computed once to construct a separate perception dataset. We recompute it continuously on the reasoning data itself and use it to schedule.
Several methods supply scheduling machinery we reuse, and each has a limit we address. Dynamics-Predictive Sampling (DPS) models each prompt's mastery as a three-state hidden Markov model with decayed Bayesian updates. It samples partially solved prompts before generating any rollouts, which cuts rollout budgets to under 30%. We extend its state tracking to three reward channels. DPS itself is restricted to one binary outcome and was evaluated mostly on text plus Geometry3k. VL-Cogito weights prompts online by rollout accuracy with a sine-shaped function inside three fixed stages. AdaCuRL bins prompts by pass rate and revisits easier bins. Its authors note that it never separates perceptual from logical failure. Puzzle Curriculum GRPO prioritizes medium-difficulty self-supervised puzzles and documents "reasoning drift" under outcome rewards. All four use a scalar difficulty measure that a title shortcut can inflate.
RuCL attacks shortcuts from the reward side, with 20 rubrics scored by an LLM judge and ordered into tiers once, before training. It constrains our design in two ways: it shows that process signals help, and it shows that a judge in the loop is expensive. Our signals need no judge. Each one is an ordinary verifiable-reward rollout under a different input.
Research Program
The build has three parts. First, an offline pass attaches a perception-only description to each training image. For charts, the description is a derendered data table, the approach Chart-based Reasoning uses with Chart2Table and Synthesize Step-by-Step uses with DePLOT. For diagrams, it is a captioner output constrained to list entities, labels, and measurements, with no inferences. Descriptions that leak the answer are rejected by string-matching the gold answer. The same tables and annotations generate the perception-probe pool of Chart101-style elementary questions.
Second, the scheduler keeps a DPS-style decayed posterior for each channel of each prompt. The image channel is updated from every training rollout. The blind channel is cheap because it has no image tokens, and we probe it with a few rollouts at each revisit. The description channel is probed only for prompts whose image-channel posterior sits in the frontier. The total rollout budget, including probes, is capped at the budget the baselines use.
Third, the controller sets $\lambda_t$ from the frontier-averaged gap and reasoning deficit. It is bounded in $[0.05, 0.5]$ so that neither pool starves.
The analysis side tracks how often each prompt's diagnosis changes category during training. The category is shortcut-solvable, perception-bottlenecked, reasoning-bottlenecked, or frontier. This measurement tells us directly whether the online part of the method earns its cost.
Baselines
The naive reference is uniform-sampling GRPO on the same data. VL-Cogito reports this vanilla GRPO setup at a 58.1% benchmark average on Qwen2.5-VL-7B, and our method must clear our own re-run of it by a margin larger than seed variance.
The strongest scheduling competitor is DPS applied unchanged to multimodal data, with its single binary outcome channel. It is the best published method for spending rollouts well. Its reported headline is 51.04% on AIME24 against 45.83% for uniform sampling on DeepSeek-R1-Distill-Qwen-7B. That result is on text, and no reported result was found on Lacuna for DPS on MathVista or MATH-Vision, so we run it ourselves.
The strongest multimodal curriculum competitors are VL-Cogito and RuCL. VL-Cogito reports 74.8% on MathVista with Qwen2.5-VL-Instruct-7B. RuCL reports a 60.06% seven-benchmark average on Qwen2.5-VL-7B trained on ViRL-39K. We reproduce both on our data, and we count RuCL's judge calls as compute.
The strongest staging competitor is From Seeing to Thinking's fixed three-stage order, given the same probe pool as its perception stage.
All methods use Qwen2.5-VL-7B, ViRL-39K plus the probe pool, identical total rollouts, identical tuning sweeps of equal size, and three seeds. The key ablations are these: our scheduler with the blind factor removed; our scheduler with $\lambda_t$ frozen at its time-averaged value; and a static version that computes all three channels once at step 0 and never updates them. Beating the field means a higher mean on the core suite than DPS and VL-Cogito at matched rollouts, together with a larger improvement on the shortcut-resistant metrics described below. It also means the static ablation lagging the online version. If the static version matches the online one, the online claim fails, whatever the headline accuracy.
Why not just…?
Why not just filter out blind-solvable prompts once, offline, and run DPS? This is the most serious threat. Shortcut solvability may be a fairly fixed property of a prompt, and if so a one-time filter captures most of the value at almost no cost. The static-diagnosis ablation settles this. The category-change statistics show whether the shortcut and gap labels actually move during training. If they rarely move, we will report that the useful contribution is the diagnosis, not its online updating.
Why not just use fixed perception-then-reasoning staging as in From Seeing to Thinking? Learning When to Look argues for the opposite order and also reports gains, which suggests neither fixed order is right for every model or dataset. We run both orders and the adaptive $\lambda_t$ on two backbones. If one fixed order beats adaptation on both, the controller is unnecessary.
Why not just add a process reward, as RuCL does? Process rewards do target shortcuts. However, RuCL's tiers are fixed before training and every rollout needs an LLM judge. Our signals cost ordinary rollouts. The matched-compute comparison, with judge FLOPs counted, shows whether the judge buys more than the diagnosis does.
Why not just spend the probe rollouts on more training rollouts? Probes cost budget. The uniform-GRPO and DPS baselines receive exactly the rollouts our method spends on probes, so any gain must survive paying for its own measurements.
Proposed Experiments and Decisive Checks
We evaluate on MathVista, MATH-Vision, WeMath, LogicVista, ChartQA, and the CharXiv reasoning subset for accuracy. Two further metrics target shortcuts. The first is blind-retained accuracy on each eval set: the share of correct answers that survive when the image is removed. We expect this to fall under our method, because more of the model's correct answers should depend on the image. The second is Chart101 elementary-reading accuracy with titles removed or perturbed, plus accuracy under the gridline and marker changes that Plot Twist shows cut PaLI-3 from 29.5% to 21.3%.
The decisive check is a joint one. The factored scheduler must beat scalar DPS at matched rollouts on these shortcut-resistant metrics, not only on headline accuracy, and the online version must beat the static version. A secondary check is whether $\lambda_t$ follows interpretable paths: rising early when the perception gap dominates, and rising again if visual forgetting appears during long-chain training, as Learning When to Look predicts. If headline accuracy rises but blind-retained accuracy does not fall, the method has only rearranged which prompts are sampled and has not closed the multimodal gap. We would report that outcome as a negative result.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
- Learning When to Look: A Disentangled Curriculum for Strategic Perception in Multimodal Reasoning
- Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models
- VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
- AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting
- Puzzle Curriculum GRPO for Vision-Centric Reasoning
- RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA