Rooted Rewards: Gating Answer Credit on Verified Perception in Multimodal RL

Vision-language models often get the hard question right while failing the easy one about the same picture. On charts, models answer multi-step questions better than they read off an axis maximum, and their accuracy collapses once the chart title stops giving the answer away (Plot Twist). Reinforcement learning on final answers makes this worse, because a lucky or shortcut answer earns full reward. We propose a simple change to that reward. Before reasoning, the model must state the visual facts the problem depends on: bar heights, axis extremes, labeled lengths. These stated facts are checked against the ground-truth data the image was rendered from. The model earns credit for a correct final answer only to the degree that its stated facts are right. A correct answer that is not "rooted" in correct perception earns little.

The idea comes from curriculum-integration research in education. That work argues that elementary skills taught as standalone drills, cut off from the advanced tasks that use them, become superficial and redundant. It recommends checking the foundational skill inside the advanced task instead. The physical-education literature documents identical sprint drills repeated across school tiers (Integrated Construction Path of PE Ideological Content). Other work argues that values and methods must grow in the "soil" of the discipline rather than be appended to it (Root, Basis, Soul), and that traditional sports taught as isolated drills lose their meaning (Wenshan minority sports case study). We take this one principle, embedded and hierarchical assessment of prerequisites, and turn it into a verifiable reward. We test whether it closes the gap between perception and reasoning better than training perception as a separate drill.

Core Claim

We claim the following: multiplying the outcome reward by the verified accuracy of the model's own stated visual facts in the same rollout will reduce shortcut solutions more than any of three alternatives at equal compute and data. The first alternative is adding the same perceptual questions as separate training items. The second is grading reasoning with an LLM judge. The third is outcome-only curriculum RL.

Concretely, for a rollout on item $i$, let $c \in \{0,1\}$ be final-answer correctness. Let $\hat g_j$ be the value the model declares for prerequisite fact $j$, and let $g_j$ be the ground truth from the image's source data. The reward is

$$ r = c \cdot s^{\gamma}, \qquad s = \frac{1}{|G_i|}\sum_{j \in G_i} \mathbf{1}\left[\,|\hat g_j - g_j| \le \tau |g_j|\,\right], $$

with the following definitions. $G_i$ is the set of prerequisite facts for item $i$, and $s$ is the fraction of them declared correctly. $\tau = 0.05$ is the tolerance, matching the 5% margin used by Chart101. $\gamma$ is a gate strength annealed from 0 (pure outcome reward) to 1 during training.

The coupling must be per rollout. If the gate instead used an image-level estimate of perceptual competence, it would multiply every rollout in a group by the same constant. Group Relative Policy Optimization (GRPO), the RL method that scores each sampled answer relative to the others for the same prompt, normalizes advantages within the group, so a constant factor would cancel and change nothing. Only rollout-level coupling lets correct perception and correct answers reinforce each other.

Relation to Prior Work

Several papers motivate this proposal by documenting the gap. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning and 31.9% to visual recognition, so both failures are common and entangled. Plot Twist shows the "harder is easier" paradox and the reliance on title cues, which is exactly the shortcut a rooted reward should penalize. GAOKAO-MM reports models that identify diagram components but fail to derive from them.

The closest methods are curriculum and process-reward RL. RuCL fights reward hacking with 20 generic rubrics, such as "Visual Presence," scored by an LLM judge and fixed in difficulty before training. We replace judged, generic rubrics with item-specific facts checked against source data, and we couple them multiplicatively to the outcome rather than adding them to it. VL-Cogito weights prompts by rollout accuracy across easy, medium and hard stages. Its curriculum is one-dimensional and outcome-only, and its own ablation shows only a small gain over vanilla GRPO (58.1% to 58.9%). That small gain is part of why we do not propose yet another difficulty schedule.

Learning When to Look rewards emitting a perception token after keywords such as "wait." That rewards the act of looking, not seeing correctly. It also argues for disentangling: text-only reasoning first, perception second. This directly contradicts the embedded-assessment principle, and our experiments test the two head on. Puzzle Curriculum GRPO trains perception on separate pretext puzzles and reports limited transfer to domain reasoning. That is the "isolated drill" failure the education literature predicts, and it is a constraint our design must beat.

Synthesize Step-by-Step supplies the method for generating data. It renders charts from templates with known tables and decomposes questions into sub-questions. We reuse that template idea to produce prerequisite facts with exact ground truth. Unlike that work, we do not feed a DePLOT table to the model at test time, so DePLOT's transcription errors cannot cascade. DCoT grounds through inference-time bounding boxes and cannot recover from a wrong first turn. P1-VL freezes the vision encoder during RL. Our gate creates a perception-specific gradient, which makes unfreezing a meaningful secondary question.

Research Program

Training data. We build a paired corpus in which every training image carries machine-checkable prerequisite facts. Charts are rendered from data tables using the template approach of Synthesize Step-by-Step, so values, extremes, orderings and differences are exact. Geometry and function diagrams are rendered programmatically from parameters, so lengths, angles, intercepts and labels are known.

A teacher model writes the main questions. Each question's prerequisite set $G_i$ is the subset of source facts its reference program actually reads, rather than a random sample. This keeps the facts "in the soil" of the problem. The model is prompted to emit a short givens block before its reasoning, and a parser extracts $\hat g_j$.

Items from ViRL-39K, which have no source data, are mixed in with outcome-only reward. This lets us check whether rooted training on part of the data improves grounding on the rest.

Model and schedule. The base model is Qwen2.5-VL-7B, the base used by RuCL and VL-Cogito. Training uses GRPO with $\gamma$ annealed from 0 to 1. The annealing is our only curriculum element, borrowed from the progressive vertical alignment idea: early on, outcome reward keeps the learning signal alive and avoids the entropy collapse P1-VL reports when rewards are near zero. Later, rooting is enforced.

Measurement. We add one measurement that education research says it lacks: a quantitative "rootedness" score. Grounded accuracy is the fraction of items where the final answer is correct and all paired prerequisites are answered correctly. The prerequisites are asked in a fresh, separate context, so the model cannot copy from its own chain.

Baselines

The naive reference point is the untrained Qwen2.5-VL-7B base model, evaluated with and without "think step by step" prompting. MATH-Vision reports that chain-of-thought can lower LMM accuracy, so both variants are needed.

The first strong competitor is outcome-only GRPO with VL-Cogito's progressive curriculum, re-run on our exact data mix with the same rollout count, steps and tuning budget. Its reported results on the shared base model are MathVista 74.8%, LogicVista 48.9%, and Geometry3K 68.7% (VL-Cogito).

The second is RuCL's rubric reward with an LLM judge, trained on ViRL-39K plus our paired corpus. It reports 60.06% average across seven benchmarks, +7.83 over the base, with +12.97 on WeMath (RuCL).

On WeMath we also compare against the Learning When to Look model's reported 54.4 (Learning When to Look). On ChartQA's human-written split, the reference is Synthesize Step-by-Step's 53.9% (Synthesize Step-by-Step). Reported numbers come from other training setups, so the decisive comparisons use our reproductions.

The ablation that isolates the key ingredient is called ungated multitask. It uses the same prerequisite facts, but as separate training questions with their own rewards, and the main question gets outcome-only reward. A second ablation is additive gating ($r = c + \lambda s$, where $\lambda$ weights the fact score), which tests whether multiplication matters. A third fixes $\gamma = 1$ throughout, which tests whether annealing matters. A fourth unfreezes the vision encoder under the full method.

All runs share data, token budget and hyperparameter search size. Beating a baseline means two things together. First, outcome accuracy must be no worse than the baseline within seed variance on MathVista, MATH-Vision, WeMath and LogicVista. Second, grounded accuracy and title-robustness must be clearly higher, by a margin exceeding three-seed variance.

Why not just…?

Why not just add the perception questions as extra training items? This is the most serious threat. It is the cheapest version of the idea, and it is exactly the ungated multitask ablation. The education literature and Puzzle Curriculum GRPO's limited transfer both predict that separately drilled skills are not recruited by the advanced task. But that prediction is not established for the case where the drill uses the same images. If ungated multitask matches the gated reward on grounded accuracy and title-robustness, the coupling claim is false and we will report it that way.

Why not just use an LLM judge to grade the reasoning, as RuCL does? A judge checking "Visual Presence" can be fooled by fluent text that mentions the image. It also costs a proprietary model call per rollout. Our check is exact and free, but only where source data exists. The RuCL reproduction will show whether exact checking buys grounding beyond what judged rubrics already deliver.

Why not just convert the image to a table with DePLOT and reason in text? Synthesize Step-by-Step shows this helps on charts but suffers cascading transcription errors. P1-VL documents an information gap for schematics whose constraints are absent from text. We include a DePLOT-plus-LLM pipeline on the chart benchmarks. We expect it to be competitive on clean bar charts and weak on scatter plots and geometry.

Why not just scale or run vanilla outcome RL longer? GPT-4o reaches only 30.39% on MATH-Vision against 68.82% for humans. GeoSQA's neural models stay near chance even with perfect text descriptions of the diagrams. Outcome-only RL is the reward-hacking regime RuCL documents. Our VL-Cogito-style reproduction serves as the long-run outcome-only control.

Proposed Experiments

Experiment 1: the decisive check. We use Chart101 (Plot Twist), where top models often fall below 25% on elementary questions, together with paired ChartQA images. The paradox gap is the difference between accuracy on complex questions and on elementary questions about the same image. We measure this gap and grounded accuracy for every arm. We also measure accuracy under neutral or misleading titles and under added grid lines, the perturbations that dropped PaLI-3 from 29.5% to 21.3%. The rooted reward should shrink the paradox gap and the title-reliance drop relative to both outcome-only RL and ungated multitask.

Experiment 2: transfer. We test transfer to items with no source data, which the rooted reward never touched. These are MathVista, MATH-Vision (broken out by subject, since Transformation Geometry is GPT-4o's weakest at roughly 18%), WeMath, LogicVista, and the science subjects of GAOKAO-MM. We report Puzzle Curriculum GRPO's reasoning-answer inconsistency rate on all of them. If gains appear only on in-distribution charts, the method is a chart tool rather than a remedy for the perception-reasoning gap, and we will say so.

Experiment 3: embedded versus disentangled. This settles the contradiction with Learning When to Look. We compare three arms on WeMath and MathVista with matched compute. The first is a text-only reasoning warm-up followed by our gated RL. The second is a perception-first warm-up on the prerequisite facts followed by outcome RL. The third applies the gate from the start with annealing. Finally, we run the gated reward with the vision encoder unfrozen. This tests whether a reward that targets perception specifically lets the encoder adapt without the instability that led P1-VL to freeze it.