Contrastive Fact Credit: Splitting Perception Blame from Reasoning Blame in Multimodal RL

Vision-language models often answer a chart or geometry question wrong for one of two different reasons. Either they misread the picture, for example by taking the wrong bar height, or they read it correctly and then reason badly from what they read. When these models are trained with reinforcement learning on a single pass/fail signal at the end of the answer, every token in a failed answer is pushed down by the same amount. The model is never told which of the two things went wrong. It can also be rewarded for right answers reached from wrong readings, which happens because question wording and chart titles often give the answer away.

We propose a cheap way to separate the two kinds of blame without bounding-box labels, a learned judge, or extra rollouts. The policy first writes down what it sees as a short list of discrete, checkable facts, such as "bar 2019 = 43" or "angle ABC = 30°", and then reasons from that list. For each training question the RL algorithm already samples many answers. We compare which facts appear in the answers that succeeded against those in the answers that failed. Facts that separate successes from failures earn perception credit. The part of the outcome that the facts cannot explain is charged to the reasoning tokens.

Core Claim

A group of sampled answers to the same image already contains enough contrast to estimate which perceptual readings matter for success. That estimate can be computed from counts at essentially zero cost. Using it to split the advantage into a perception part and a reasoning part should reduce perception errors specifically, measured by probes with known ground truth, and should reduce cases where the right answer comes from the wrong reading. It should do this at no more compute than standard group-based RL.

The mechanism borrows the central move of Align-RUDDER. Align-RUDDER aligns a few successful trajectories into a position-specific scoring profile and redistributes a delayed reward onto conserved events. It struggles in two places. It needs continuous states clustered into a discrete alphabet of events, and it assumes all successful trajectories follow one ordering. Written perceptual facts remove both problems. They are discrete by construction, and treated as an unordered set they do not care whether two correct solutions reason in different orders.

Relation to Prior Work

Several papers motivate the problem. MATH-Vision annotates GPT-4V failures and finds 42.2% reasoning errors against 31.9% vision recognition errors. Both error types are therefore large, and they call for different corrections. Plot Twist shows the "harder is easier" paradox: models do better on complex chart questions than on elementary reading, because text heuristics carry them. Outcome-only reward will reinforce exactly that behavior. GeoSQA shows that even perfect human-written diagram descriptions leave text models near chance. Reasoning credit must therefore be learned separately rather than assumed to follow from correct perception.

Four methods attack step-level credit in multimodal RL, and each fixes something we avoid.

  • SketchVL draws marks on the image and scores each drawing with a separately trained process reward model. We use no learned judge, so we have no judge bias to be hacked.
  • POLIA scores the bounding boxes in the model's reasoning against ground-truth boxes. We need no spatial labels, which makes the method usable on exam diagrams and math figures where no boxes exist.
  • MBPO branches reasoning trees at <look> tokens and compares sibling branches. It still pays tree-expansion cost. Its sibling comparisons also measure the value of a whole continuation rather than which specific reading drove success.
  • ProcessThinker estimates step value by running several continuations from each step. This gives an unbiased value estimate at $O(S\times M)$ extra rollouts per trajectory, where $S$ is the number of steps and $M$ the continuations per step.

Our estimator is biased where theirs is unbiased, but it costs nothing beyond the group already sampled. Measuring that trade-off is part of the program.

Two further papers shape the design. The survey A Survey of Temporal Credit Assignment in Deep RL insists that credit should track an action's influence on the outcome, not merely its value. This motivates contrasting successes against failures rather than rewarding consensus. A reading that appears in every answer has high value but no influence on which answers succeed. The hierarchical view in On Credit Assignment in Hierarchical RL treats high-level choices as skip connections for reward. It justifies giving the perception block its own credit channel instead of letting the terminal reward diffuse through every reasoning token.

Research Program

The policy is trained to emit a perception block before its reasoning. The block is a list of typed facts: an entity key (a series and category, a labeled segment or angle, or a legend item) and a value (a number, unit, or category). Canonicalizing these facts is the main engineering risk. We handle it with deterministic rules: numeric values match within a relative tolerance, keys are normalized against text read from the image, and facts that cannot be parsed are dropped. The format is taught with a short supervised warm-up so that it is not itself the source of gains.

For a question with $G$ sampled answers, let $N_s$ and $N_f$ be the number of successes and failures. For each canonical fact $f$, let $n_s(f)$ and $n_f(f)$ count the successful and failed answers that contain it. With a smoothing constant $\alpha$, fact $f$ receives a log-odds score

$$ s(f)=\log\frac{n_s(f)+\alpha}{N_s+2\alpha}-\log\frac{n_f(f)+\alpha}{N_f+2\alpha}. $$

This is the scoring-matrix idea from Align-RUDDER applied to sets of facts rather than aligned sequences.

For answer $i$ with fact set $F_i$, the perception tokens receive the group-normalized mean of $s(f)$ over $f\in F_i$. The reasoning tokens receive a residual advantage $A^{ded}_i = R_i - \hat p_i$, where $R_i\in\{0,1\}$ is the outcome and $\hat p_i$ is a leave-one-out estimate of success from $F_i$. We compute $\hat p_i$ from the scores of the other $G-1$ answers so that an answer cannot explain its own outcome.

Two consequences follow. A correct answer built on low-scoring facts gets a small perception reward, which directly targets right-for-the-wrong-reason answers. A failed answer whose facts look like those of successes has its penalty placed mostly on the reasoning tokens.

When a group is all-success or all-failure, the scores are zero. The method then falls back to plain outcome advantage. We report how often this happens, since it is exactly the advantage-collapse regime MBPO documents.

We will build this on Qwen2.5-VL at 3B and 7B, the base models used by SketchVL and MBPO, trained on GRPO with chart data (ChartQA and PlotQA training splits) and visual math data. The analysis has three parts:

  • whether fact scores agree with the true correctness of facts, using the data tables that come with ChartQA and PlotQA as held-out ground truth that training never sees;
  • how perception and reasoning advantages evolve over training;
  • whether the entropy rescaling of EMPG composes with the split.

Baselines

The naive reference is the base model prompted with the same perceive-then-reason format and no RL. It tells us how much the format alone contributes.

The main controlled baseline is vanilla GRPO with the same format, data, group size, and training steps. Every token receives the outcome advantage. This isolates the credit split from the structured output. The key ablation keeps everything identical but replaces the contrastive score with consensus among successes only, dropping the failure term. This tests whether influence rather than popularity is what matters.

The strongest existing competitor on the reasoning benchmarks is MBPO. It reports 87.7% on ChartQA and 52.6% on MathVerse with Qwen-VL-7B, and it beat GRPO by up to 8.7 points at matched TFLOPs. We will run it on our data with the TFLOPs budget equalized. Beating it means matching its accuracy while being clearly better on perception probes, or exceeding its accuracy at equal compute.

The most direct competitor for the credit estimator itself is single-boundary continuation rollouts in the style of ProcessThinker. We sample $M$ reasoning continuations from each perception block and use their success rate as the block's value. This is the unbiased version of what we approximate. We compare it at equal total rollouts, giving the contrastive method the extra budget as larger groups, and also at unequal compute to measure the accuracy gap.

On chart reasoning, SketchVL reports 58.64 on EvoChart-QA at 7B against 40.32 for VLM-R1, and a 15.32-point ChartQA gain at 3B. We compare against these reported figures where our setup matches theirs. POLIA is a label-using upper reference on VSR, where it reports 81.3%. Coming close to it without boxes would count as a strong result.

Why not just…?

Why not just run vanilla GRPO with the perceive-then-reason format? Structured output may by itself make the model read more carefully. That is why format-matched GRPO is the main controlled baseline. The evidence against it being sufficient is that POLIA finds outcome-only GRPO leaves reasoning hallucinations, and MBPO finds its advantages collapse. If format-matched GRPO equals our method on the ground-truth fact-accuracy probe, the credit split adds nothing and we will say so.

Why not just use continuation rollouts from the perception block? This is the real threat. It estimates exactly the quantity we approximate, and it does so without bias. Our argument is cost: one boundary with $M$ continuations multiplies rollout compute by roughly $M$. The matched-compute comparison settles it. If spending the same budget on continuations beats spending it on larger groups for contrast, the right method is the rollout one, and our contribution shrinks to the diagnostic analysis.

Why not just reward agreement among sampled readings (self-consistency) or use a model judge? Agreement rewards the popular reading. The popular reading is often the one text heuristics suggest; Plot Twist reports accuracy falling to about 30% under misleading titles. Judges invite gaming: ADMIRE reports agents producing judge-pleasing but useless steps. The consensus-only ablation and the misleading-title evaluation test the first point directly.

Why not just fine-tune on synthesized perception tables, as in chart de-rendering? Chart-based Reasoning shows de-rendering pretraining helps, but it needs table supervision. ProcessThinker finds supervised fine-tuning teaches format but not task success. Our method applies where no tables exist, such as MATH-Vision diagrams. We will still report a table-supervised fine-tuning run on ChartQA as a label-using reference.

Proposed Experiments

Accuracy will be measured on ChartQA, EvoChart-QA, MathVista, MathVerse, and MATH-Vision. On MATH-Vision the reported reference is GPT-4o at 30.39% against 68.82% for humans. The decisive checks, however, target perception directly, and there are four of them.

First, on held-out ChartQA and PlotQA charts, we score every emitted fact against the underlying data table. The proposal predicts higher fact accuracy than format-matched GRPO at equal answer accuracy. We also compute the rate of right-answer-with-wrong-fact cases, which should fall.

Second, on the Chart101 elementary perception probes from Plot Twist, including its misleading-title condition, the method should gain more than GRPO. This is where outcome-trained models are weakest; the file reports ChartLlama at 4.2% on scatter plots. No reported result was found on Lacuna for RL-trained models on Chart101.

Third, we will hand-annotate 200 MATH-Vision failures per model with the paper's error taxonomy. The proposal predicts that the vision-recognition error fraction drops more than the reasoning fraction relative to GRPO. A proportional drop would argue against the claim that the split targets perception.

Fourth, we will plot the correlation between the contrastive fact score and true fact correctness as a function of group size. The estimator's bias should shrink as groups grow. The group size at which it matches continuation rollouts tells practitioners when the cheap estimator is enough.