Rewarding What the Model Saw: Separate Perception and Reasoning Rewards for Compact Chart-and-Diagram Reasoners

When vision-language models are trained with reinforcement learning to reason in long chains, they get better at math over images and worse at looking at the image. More Thinking, Less Seeing? measured this directly. As reasoning chains grow, the model's attention drifts from the image toward its own generated text. Models trained with supervised fine-tuning followed by reinforcement learning lost the most perceptual grounding. The cause is plausibly the reward itself. Current training scores only the final answer, and sometimes format and length as well. A chain that misreads a bar height but lands on the right answer earns the same reward as a chain that read everything correctly. The model is never told what it saw.

We propose to fix this inside training, for a compact (7B) model specialized to scientific charts and geometry diagrams. We render the training figures from programs, so every value in every figure is known exactly. The policy must state its visual observations as tagged claims inside its reasoning. A checker scores those claims against the rendering program, with no neural judge involved. That perception score is kept as a separate reward and normalized separately from answer correctness. It is never summed into a single number, following the decoupled normalization of GDPO. The central bet is that a verifiable, separately normalized perception reward can stop grounding from eroding during long-chain RL without costing reasoning accuracy. We also want to learn whether that works better than the simpler fixes a reviewer would reach for first.

Core Claim

Grounding decays during multimodal RL because perception is invisible to the reward. Perception can be made visible and checkable at no annotation cost when the figure comes from a program. The resulting signal must stay a separate dimension of the reward. Otherwise it collapses into the answer reward, which is the failure GDPO calls "reward signal collapse" in text-only multi-reward RL.

The prediction is specific. The same compact model, trained on the same prompts with the same compute, should show three things relative to outcome-only RL. It should keep higher accuracy on pure-perception items as its chains lengthen. It should reach a higher reasoning–hallucination trade-off curve. It should match or beat outcome-only RL on reasoning accuracy.

The closest existing work each covers one piece and misses the conjunction:

  • GDPO decouples rewards but, by its authors' own statement, leaves vision-language models open.
  • URSA trains a step-level process reward model on injected misreadings. However, it collapses each step into a single confidence score, so perception and algebra errors are indistinguishable.
  • Perception-Judge trains a judge to catch perceptual errors in explanations. It has never been used inside a training loop, and it is still a neural model open to exploitation by the policy.
  • MIRAGE scores reasoning chains on several dimensions, but only as an LLM-judged evaluation (its Logos training uses curriculum and reward filtering, not separate perception rewards).

Research Program

Data. We build the training environment first. It contains parametric chart generators (bar, line, scatter, and multi-panel scientific plots with axes, legends, error bars, and log scales) and geometry diagram generators. Each generator emits three things: the image, a question, a solution program that computes the answer, and the set of visual quantities that solution program reads. For example, "the value of series B at x=3" or "angle ABC as drawn."

This addresses a limitation the file names for expert visual domains: charts and diagrams have no execution compiler, so rewards have come from biased neural judges. Here the rendering program serves as the compiler. VeriSciQA showed that cross-modal verification matters: only 3% of candidates survived its filters, and fine-tuning on the survivors gave +2.05% on average. We keep its verified QA pairs (datajuicer/VeriSciQA) as a held-out, real-figure transfer test rather than as training data.

Policy and rewards. The policy is Qwen2.5-VL-7B. It reasons freely but must wrap each visual claim in an observation tag, as a quantity-value pair. The perception reward $r_{\text{per}}$ has two parts:

  • Precision is the fraction of stated observations that match the rendering program within the figure's resolution tolerance.
  • Recall is the fraction of the solution's required quantities that the chain actually observed.

The recall term closes the obvious loophole of stating nothing or stating only trivial facts. Observations are checked wherever they appear in the chain, not only at the start. This matters because drift happens late in long chains. The correctness reward $r_{\text{acc}}$ is exact-match on the final answer, and a format reward $r_{\text{fmt}}$ checks the tag structure.

Combining the rewards. For a group of $G$ rollouts on one prompt, the advantage of rollout $i$ is

$$A_i = \sum_{k \in \{\text{acc},\,\text{per},\,\text{fmt}\}} w_k \,\frac{r_{i,k} - \mu_k}{\sigma_k + \epsilon},$$

where $r_{i,k}$ is rollout $i$'s reward on dimension $k$, $\mu_k$ and $\sigma_k$ are that dimension's mean and standard deviation within the group, and $w_k$ are fixed weights. This is followed by batch-wise normalization of $A_i$, as in GDPO.

The point of normalizing each dimension separately is concrete. When every rollout gets the answer right, $r_{\text{acc}}$ has zero variance and contributes nothing. Differences in perceptual accuracy still produce a gradient. That is exactly the situation where outcome-only GRPO has vanishing advantages, which VL-Rethinker had to patch with sample replay. We train with RL only and no supervised warm-up, because the file reports that supervised fine-tuning followed by RL degraded grounding most.

Analysis. Beyond benchmark scores, we track three quantities over training:

  • the fraction of attention mass on image tokens as a function of chain position, which is the measurement used in More Thinking, Less Seeing;
  • observation precision on held-out rendered figures;
  • whether a linear probe for answering without the image, of the kind in Mirage Probes, fires less often after training.

The probe test also gives a training-time causal check on the probe direction, which that paper left open.

Baselines

The naive reference point is the untrained Qwen2.5-VL-7B base with its native prompting. MIRAGE reports 68.2% MathVista accuracy and 28.8% MIRAGE accuracy for this model.

The main controlled competitor is outcome-only GRPO on the identical rendered prompts, with the same rollout count, steps, learning-rate sweep, and generation budget. It is the strongest competitor because pure RL is what the file reports grounds best: Ocean-R1-7B, an RL-only model, reached the highest RH-AUC reported, 0.63, in More Thinking, Less Seeing?. We also evaluate Ocean-R1-7B itself on RH-Bench as an external anchor. The proposal beats this competitor only if it improves the perception half of RH-Bench and RH-AUC while staying within noise or better on reasoning accuracy, measured over three seeds.

The ablation that isolates the key ingredient is summed-reward GRPO. It uses the same three rewards, including the verifiable perception reward, but adds them before a single group normalization. If this matches the decoupled version, then decoupling is not what matters and the claim reduces to "add a perception reward."

A second ablation isolates verifiability. It replaces $r_{\text{per}}$ with a perception score from a Perception-Judge-style 7B judge. That judge's reported perceptual error detection accuracy is 85.7%, against 69.5% for its Qwen2.5-VL-7B base. The ablation runs under the same decoupled normalization and compute.

A third ablation is perception as separate prompts. Perception-only questions about the same rendered figures are mixed into the RL prompt pool with outcome reward only. This tests whether in-chain observation checking is needed at all.

Against published systems, we take the following numbers as reported. The strongest compact model for multimodal math is URSA-8B-PS-GRPO at 31.5% on MathVision (URSA). Logos-7B reports 72.3% on MathVista and 37.1% accuracy on MIRAGE (MIRAGE). Functional Attention Control, an inference-time reweighting of perception and reasoning attention heads, reports an average +4.2-point gain across five benchmarks. We apply FAC to both our model and the outcome-only GRPO model, to test whether the gains stack.

Our training data differ from URSA's million-scale corpus. We therefore treat beating Logos-7B on MIRAGE and matching URSA on MathVision as the published-number targets, not as controlled comparisons.

Why not just…?

Why not just run outcome-only RL, since pure RL already grounds well? This is the real threat, and the proposal could lose to it. The evidence that it falls short is indirect. Even the best RL-only model reported sits at an RH-AUC of 0.63, which leaves headroom. MATH-Vision attributes 31.9% of GPT-4V errors to misrecognition, and an answer-only reward is silent on misrecognitions that happen to cancel out. The controlled GRPO baseline settles this question directly. If the perception half of RH-Bench does not improve over it, the thesis is wrong.

Why not use a trained judge such as Perception-Judge as the perception reward? It is attractive because it transfers to real figures without rendering programs. However, it detects only 85.7% of perceptual errors on its own test set, and the file documents that multimodal judges hallucinate errors and anchor on the response they are grading. Under optimization pressure, a policy may learn to exploit exactly those blind spots. The judge-reward ablation measures this through a specific signature: judge-scored perception rises while rendering-verified observation precision falls.

Why not just add perception questions to the training mix? It is cheaper and needs no observation tags. Its weakness is that drift is a within-chain phenomenon. Answering "what is the bar height" well in isolation does not guarantee the model re-reads the chart at step twelve of a derivation. The separate-prompts ablation, analysed by chain position, shows whether that concern is real.

Why not just apply FAC at inference? FAC is training-free and cheap, adding about 9% latency. But it uses fixed layer boundaries and gains, and it cannot change what the model learned to rely on. We treat it as complementary and report stacked results. If FAC applied to outcome-only GRPO matches our model, the training-time signal is not earning its cost.

Proposed Experiments and Decisive Checks

Benchmarks. Evaluation spans five kinds of test:

  • RH-Bench, both halves, with RH-AUC across token budgets. This is the primary grounding-versus-reasoning measure.
  • Reasoning: MathVista, MathVision, and MIRAGE, including MIRAGE's spatial category, where the authors found scaling barely helps.
  • Real scientific figures: the VeriSciQA pairs.
  • False-premise robustness: HaloQuest (open-source VLMs below 36% reported).
  • In-domain perception: observation precision and recall on held-out rendered families, reported as separate dimensions rather than one score.

Decisive checks. Four results decide the outcome:

  1. Grounding does not decay. Relative to outcome-only GRPO, the image-attention fraction late in the chain stays flat instead of falling, and RH-Bench perception accuracy does not decline as the reasoning budget grows.
  2. Decoupling matters. Summed-reward GRPO underperforms the decoupled version on the perception dimension after training has matured. This mirrors GDPO's observation that summed rewards collapsed near 400 steps.
  3. Verifiability matters. The judge-reward variant shows a growing gap between judge-scored and rendering-verified perception.
  4. The signal transfers. Gains on rendered training charts carry over to VeriSciQA's real figures. Without transfer, the method only teaches the model to read our renderer.

If the first check fails against outcome-only GRPO, we report a negative result. That outcome would itself be informative: it would mean answer-only RL is enough to keep compact models looking at the figure.