Factored Credit for Reading and Reasoning in Multimodal Reinforcement Learning

When a vision-language model answers a chart or diagram question, it has to get several things right at once. It must read three or four values off the image correctly, and then it must combine them with the right operations. Current reinforcement learning for these models rewards only the final answer. The chance that a sampled answer is fully correct is roughly the product of the chances that each reading and each step is correct, so the learning signal fades exponentially as questions need more pieces. This is the same problem that "combinatorial action spaces" (decisions made of many parts chosen jointly, whose number of combinations explodes) pose in classical reinforcement learning. Bandit theory has a known remedy for it: observe the outcome of each part separately.

We propose to import that remedy. The model writes its answer in two parts, a structured "reading" block of extracted visual values and a "derivation" block that computes from them. On charts rendered from known data tables, each reading can be checked against the table, and the derivation can be checked by asking whether it computes correctly from the model's own readings. Each span of the output then receives credit from its own verified outcome rather than from a single scalar. The claim is specific and testable. If the bottleneck is the multiplicative fading of the outcome signal, then the advantage of factored credit over outcome-only training should grow with the number of visual values a question requires. Separately, the per-component outcomes give a direct measurement of whether failures come from perception or from reasoning, which the literature currently disputes.

Core Claim

Factored-Reward Bandits with Intermediate Observations studies actions made of $d$ components whose reward is a product of component outcomes. It shows that a learner seeing only the final product suffers regret growing exponentially in $d$, while a learner that also sees each component's intermediate outcome achieves regret linear in $d$, with matching lower bound $\Omega(\sigma d\sqrt{kT})$. A multimodal reasoning rollout fits this shape closely. Write $p_i$ for the policy's probability of getting component $i$ right, where the components are the readings $1,\dots,d$ and the derivation $0$. The probability that a rollout earns a positive outcome reward is

$$P(\text{correct}) \approx \prod_{i=0}^{d} p_i ,$$

so for a group of $G$ rollouts in Group Relative Policy Optimization (GRPO, which normalizes each rollout's reward against the other rollouts sampled for the same prompt), the chance that the whole group is uninformative grows rapidly with $d$ when the $p_i$ are moderate. Our method replaces the single advantage with a per-span advantage. Let $x_i^{(g)}\in\{0,1\}$ be the verified outcome of component $i$ in rollout $g$. Every token in span $i$ receives

$$A_i^{(g)} = \frac{x_i^{(g)} - \mathrm{mean}_g\, x_i}{\mathrm{std}_g\, x_i + \epsilon},$$

and the derivation outcome $x_0$ is defined as "the gold question program applied to the model's own extracted values equals the model's answer." This definition credits correct reasoning even when the reading was wrong, and it penalizes a lucky final answer reached from wrong readings. The new element is not process supervision in general. It is credit split along the same factorization that makes the action space combinatorial, with each factor verified symbolically rather than judged by a learned model.

Positioning

Three groups of prior work frame the idea. The first motivates it. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 42.2% of GPT-4V errors to reasoning and 31.9% to recognition. Perception-Aware Policy Optimization for Multimodal Reasoning (PAPO) instead finds 67% of failures of RL-trained models are perceptual. Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details shows models answering complex chart questions better than elementary readings, and shows that accuracy collapses from 79% to around 30% when helpful titles are removed. That pattern means outcome rewards can be earned without reading at all. Our factored outcomes are exactly the instrument that settles this dispute on a given model, because they log perception and derivation correctness separately for every rollout.

The second group supplies the closest training methods, and our design differs from each. PAPO maximizes the divergence between outputs on clean versus masked images. CFPO: Counterfactual Policy Optimization for Multimodal Reasoning penalizes answers that stay the same when salient visual tokens are suppressed. Both push the model to depend on the image, but neither checks whether what it read is correct, and both still assign one scalar advantage per rollout. Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models uses outcome-only GRPO with length control. Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning uses process-level rewards from a verifier that is the same model, so its intermediate signal is itself subject to the perception errors it is meant to catch. Our intermediate signal comes from ground-truth tables and symbolic re-execution.

The third group supplies the structure. Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs pretrains on chart-to-table conversion and uses program-of-thought, which is supervised learning on the same decomposition we exploit with RL. Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA generates template questions with executable programs, which is the data source that makes $x_0$ computable. Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare constrains us. Its additive factorization is exact only when components barely interact, and it otherwise trades bias for variance. In our setting the derivation genuinely depends on the readings, which is why we condition $x_0$ on the model's own readings instead of treating components as independent.

Research Program

We first build a controllable benchmark. Charts are rendered from random data tables, and template questions are generated with explicit programs whose number of required readings $d$ ranges from one to eight. For each question we know which cells are needed, so the reading block can be scored cell by cell, with a 5% tolerance following the range-based metric of Chart101. The output format is a fixed schema: a list of (series, category, value) triples, followed by a short derivation ending in an answer. An unparseable reading block receives $x_i=0$ for all readings.

We then train Qwen2.5-VL-7B, the base used by Agent0-VL, with GRPO in four arms. The arms are outcome-only, outcome plus PAPO loss, summed shaped reward, and factored per-span advantage. They share data, group size, rollout budget, and tuning budget. Training mixes the synthetic set with ChartQA and PlotQA training charts that have underlying tables. On items without a gold program, only reading components and the final answer are verified. Finally, we transfer to benchmarks without tables or programs, to test whether the reading habit survives without reading supervision.

The analysis tracks per-component accuracy $p_i$ over training. If the factorization argument is right, the fraction of all-zero or all-one GRPO groups (zero-gradient groups) should be much higher for outcome-only training at large $d$, and factored training should raise reading accuracy first, followed by derivation accuracy.

Baselines

The naive reference point is the base Qwen2.5-VL-7B prompted with the same reading-then-derivation format and no training. It tells us how much any RL contributes and whether the schema alone helps.

The first trained baseline is outcome-only GRPO in the Vision-R1 style, run on identical data, rollouts, and hyperparameter search budget. It is the default recipe for RLVR in vision-language models, and it is the arm whose failure at large $d$ our claim predicts.

The strongest competitor on the specific question of grounding is GRPO with the PAPO loss, optionally with CFPO's counterfactual term. These are the most direct published attempts to stop multimodal RL from ignoring the image. PAPO reports overall gains of 4.4% to 17.5% over GRPO across eight benchmarks and a 30.5% reduction in perception errors. CFPO reports a further 1.32% to 2.13% over PAPO. We use their published loss weights and give them an equal tuning budget. Beating them means higher answer accuracy on ChartQA and the high-$d$ synthetic split, and higher Chart101 collective accuracy, at matched rollout count.

The strongest absolute numbers on charts come from supervised pipelines. ChartPaLI-5B with program-of-thought reports 81.3% relaxed accuracy on ChartQA, against 80.8% for Gemini Ultra and 78.5% for GPT-4V, according to Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs. We also report a matched supervised baseline: the same base model fine-tuned on chart-to-table plus program traces from our data, and then given outcome-only GRPO. We do not claim to beat 81.3% with a 7B model trained on modest compute. The relevant comparison is the matched one, and the published number serves as context.

The key ablation is the summed shaped reward. It uses the same verified component outcomes, but adds them into one scalar per rollout that is applied to all tokens. This isolates the per-span advantage, our actual ingredient, from simply having more reward information. A second ablation drops the own-readings conditioning of $x_0$ and scores the derivation against the gold answer. This tests whether separating reasoning credit from reading credit matters.

Why not just…?

Why not just add the intermediate checks as a shaped reward? This is the most serious threat. Summing component outcomes into one scalar already gives informative gradients when the final answer is wrong. It may recover most of the benefit, in which case the per-span structure is unnecessary. The difference is in where credit lands. A rollout with perfect readings and a wrong derivation gets a middling scalar that is spread over the readings too, so correct reading tokens are partially punished. The summed-reward ablation decides this directly. If it matches factored credit across all $d$, we will report that the information, not the factorization, is what matters, and the contribution shrinks to the verification scheme.

Why not just do supervised chart-to-table training, as ChartPaLI does? Supervised extraction teaches reading on the training distribution, but it does not teach the model to use its own readings rather than title cues. Chart101's shortcut results suggest that is the failure that persists. The matched SFT-then-GRPO baseline tests this. Our prediction is that SFT yields similar reading accuracy in isolation but lower "derivation-on-own-readings" consistency, and a larger accuracy drop when titles are made misleading. If SFT matches us on both measures, the RL component adds little.

Why not just use PAPO or CFPO? These losses make outputs sensitive to the image. Sensitivity is necessary but not sufficient, because a model can depend heavily on the image and still misread it. PAPO's own residual perception errors show room remains. Our arm can also be combined with PAPO. If the combination beats each alone, the two address different failures. If PAPO alone matches us on the high-$d$ split, the exponential-fading story is wrong.

Why not just use a learned process reward model or a self-verifier, as Agent0-VL does? A learned verifier is available on unannotated data, which is a real advantage. However, it shares the perception weaknesses of the policy, so on charts the reward may be noisier than the error it corrects. We will compare against a self-verification process reward on the same synthetic data, where ground truth lets us measure the verifier's own accuracy on reading claims.

Proposed Experiments

The decisive experiment is the $d$-sweep on the synthetic benchmark. We plot answer accuracy and the zero-gradient-group fraction against required readings $d$, for each arm, at a fixed rollout budget. The factorization argument predicts that the gap between factored and outcome-only training widens with $d$, and that the gap between factored and summed rewards is small at $d=1$ and grows with $d$. A flat gap across $d$ would falsify the central claim even if average accuracy improved.

Real-data evaluation covers four benchmarks. On ChartQA we report relaxed accuracy, separately for human-written and machine-generated questions, since the human split is where Synthesize Step-by-Step raised accuracy from 38.2% to 53.9%. On PlotQA the reported best is 92.89%. On Chart101 we report collective accuracy, below 25% for top models, under three title conditions: original, absent, and misleading. A model that truly reads should lose much less under misleading titles than outcome-trained arms. For transfer beyond charts, we evaluate on MathVista and MATH-Vision without any reading supervision. Vision-R1-7B reports 73.5% on MathVista and GPT-4o 30.39% on MATH-Vision. The question is whether factored training on charts preserves or improves these relative to the matched outcome-only arm, not whether it surpasses those systems.

Finally, we use the per-component logs to address the perception-versus-reasoning tension. For each arm, we report the share of wrong answers with correct readings and wrong derivation, against wrong readings. We compare these shares with the MATH-Vision and PAPO error breakdowns. We expect the dominant error type to depend on $d$ and on training method, which would reconcile the two camps instead of siding with one.