Lock What the Model Read Correctly: Self-Critical Branching for Chart Reasoning

Vision-language models often read a chart correctly and then reason about it wrongly, or read it wrongly and still land on the right answer by exploiting the title. Reinforcement learning is the standard fix for multi-step reasoning. It samples whole answers, scores only the final result, and pushes the model toward higher-scoring samples. That process has no way to tell which half failed. It also routinely throws away progress. A sampled answer re-draws every token, including numbers the model had already read off the chart correctly, much as a naive fuzzer (a tool that randomly mutates test inputs to find program bugs) keeps flipping the bytes that got its input past a validity check.

We propose a training method that borrows the fuzzer's fix. The model's own greedy answer serves as the reference. Its chart readings are checked against the chart's underlying data table. Readings that pass are locked, and exploration re-samples only the unverified readings or the reasoning that follows them. Each branch is scored against the greedy reference, the "self-critical" baseline of Self-Critical Sequence Training for Image Captioning. The resulting advantage is credited only to the tokens that were actually changed. Put in two sentences: keep what the model verifiably read right, re-sample the rest, and compare each branch to the model's own best guess. This gives separate learning signals for perception and for reasoning, and it keeps producing nonzero updates even after greedy accuracy is high.

Core Claim

On chart and plot question answering, where ground-truth data tables exist, we claim that verified-lock self-critical branching beats whole-sequence group RL at a matched token budget. We expect three measurable effects. Final accuracy should be higher. Fewer answers should be correct for the wrong reasons, meaning correct final answers built on wrong readings. The fraction of training prompts that yield zero gradient should be smaller.

The mechanism, not branching alone, is the contribution. Prefix branching for credit estimation is a known general idea, and we do not claim it. What is new here is using an external validity check, agreement with the chart's data, to decide which segments are frozen. This is the seed-mask logic of MuFuzz, which locks the input bytes that satisfied a program guard so that later mutations explore deeper code. We combine it with a self-critical greedy baseline, so the perception and reasoning segments receive separate advantages.

Motivation and Prior Work

Three findings motivate the proposal.

The first is error analysis on MATH-Vision. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset finds that 42.2% of GPT-4V's errors are reasoning errors made after correct recognition, against 31.9% recognition errors. A single final-answer reward therefore routinely punishes correct readings.

The second is evidence of shortcuts. Plot Twist shows accuracy near 79% when a chart's title leaks the answer and around 30% when it does not. The same paper reports scatter-plot reading accuracy as low as 4.2%. Final-answer rewards therefore also reinforce wrong readings that happen to reach correct answers.

The third concerns step-by-step output. MATH-Vision reports that chain-of-thought prompting hurts several models, which suggests that longer intermediate text offers more places to corrupt a visual fact.

Two recent papers apply RL to exactly this setting and supply our strongest baselines. Point-RFT first fine-tunes a model to interleave reasoning with pointed chart coordinates. It then applies group-relative policy optimization (GRPO), which scores a group of full samples per question against their mean. Its rewards are answer correctness and format adherence, and it reaches 90.04% on ChartQA. Diving into Self-Evolving Training for Multimodal Reasoning (M-STaR) identifies a "saturation paradox": as greedy accuracy rises, exploration collapses. Its remedy is to adapt the sampling temperature globally.

Both methods re-sample entire trajectories. Neither checks individual perceptual tokens against ground truth. Neither assigns credit separately to perception and reasoning. Our method changes what is sampled rather than how hot the sampler runs.

The fuzzing literature supplies the method. MuFuzz locks bytes behind passed guards. DIE goes further, restricting mutations to replacements that preserve type and structure. Driller quantifies why blind mutation cannot re-synthesize exact constants: a 32-bit value has a $2^{-32}$ chance per random attempt. A chart reading of "1,847" that has already been verified is the reasoning-chain analogue of such a constant.

Self-Critical Reasoning for Robust VQA shares a name and a goal, penalizing answers that use visual evidence spuriously. It is a discriminative gradient-sensitivity loss rather than sequence-level RL, so it motivates the shortcut metric we report but does not supply the method.

Method

The model is trained to answer in two segments. The first is a reading block $p$: a list of (label, value) entries taken from the chart. The second is a reasoning block $r$: text plus short executable arithmetic, in the style of the code rationales in Synthesize Step-by-Step. A brief supervised warm-up teaches this format. Training draws on ChartQA and PlotQA, whose charts ship with data tables. Each reading entry is checked against the table using the 5% relative-error margin that Chart101 uses. An entry is marked verified if it matches the table and unverified if it does not match or cannot be found.

For each question $x$, the model first decodes greedily to produce $\hat{y} = (\hat{p}, \hat{r})$ with reward $R(\hat{y})$, where $R$ is answer correctness. It then spends the rollout budget on two kinds of branch.

The first is a reasoning branch. Every verified entry of $\hat{p}$ is locked, any unverified entries are kept as-is, and only the reasoning is re-sampled: $r^s \sim \pi_\theta(\cdot \mid x, \hat{p})$.

The second is a perception branch. Verified entries are locked and unverified entries are re-sampled, which is the direct analogue of a byte mask. Reasoning is then greedily re-decoded from the new readings.

Each branch receives a self-critical advantage against the greedy reference, and gradients flow only through the tokens that branch re-sampled:

$$ A_r = R(\hat{p}, r^s) - R(\hat{p}, \hat{r}), \qquad A_p = \big[R(p^s, \hat{r}(p^s)) + \lambda V(p^s)\big] - \big[R(\hat{p}, \hat{r}) + \lambda V(\hat{p})\big]. $$

Here $V(\cdot)$ is the fraction of reading entries that pass verification, $\hat{r}(p^s)$ is greedy reasoning conditioned on the new readings, and $\lambda \ge 0$ weights reading accuracy.

The reasoning advantage compares reasoning with perception held fixed. This paired comparison should have lower variance than comparing whole samples whose readings also differ. When greedy accuracy is high, whole-sequence samples often collapse onto the greedy answer and give zero advantage. Branches at unverified readings, or at reasoning steps, keep producing diverse samples in exactly the places that remain uncertain.

On benchmarks without data tables, such as MathVista and MATH-Vision, we only evaluate. We do not train there. This keeps the verifier honest.

One obvious hack is for the model to emit only easy, verifiable readings. Two things guard against it. The answer reward still dominates. We also monitor the rate at which the reasoning cites values that are absent from the reading block.

Research Program

We first build the format warm-up data. This is inexpensive because targets can be generated from the tables themselves. We then implement branching on top of a standard GRPO trainer, which keeps every other component identical across arms.

All arms use one open base model. The candidates are Phi-3.5-Vision or MiniCPM-V-2.5, since M-STaR reports MathVista trajectories for both. The file does not state Point-RFT's base model, so we will not pretend to reproduce its absolute numbers. Instead we re-run its recipe on our base.

The central analysis measures, over the course of training, three quantities: the fraction of zero-advantage prompts, the variance of the reasoning-segment gradient, and the split of final errors into reading errors and reasoning errors.

Baselines

The naive reference point is the base model after the same format warm-up, with no RL. Point-RFT's format-fine-tuned model scored 70.88% on ChartQA, which shows how much headroom RL can add over imitation alone.

The strongest existing method is whole-sequence GRPO with answer and format rewards, run as Point-RFT describes. Point-RFT reports 90.04% ChartQA accuracy for this recipe. It reports 83.92% for a text-only RFT variant, and an out-of-domain average rising from 35.94% to 53.16% across CharXiv, PlotQA, IconQA, TabMWP and Counting. We re-run this recipe on our base model, warm-up data, question pool and optimizer. The rollout budget is matched in total generated tokens, not number of samples, because our branches are shorter than full samples. Hyperparameter search budgets are also equal. Beating it means higher ChartQA relaxed accuracy on the human-written split and a higher out-of-domain average, over at least three seeds, with confidence intervals that do not overlap.

The second strong competitor is M-STaR's exploration fix. We add its temperature schedule, driven by Reward-Pass@2, to the GRPO baseline. M-STaR reports MathVista gains of 52.8% to 59.5% for MiniCPM-V-2.5 and 49.3% to 54.5% for Phi-3.5-Vision. We can check our re-run against the matching figure.

Four ablations isolate the key ingredient. The first, unverified locking, locks the greedy readings regardless of correctness. The second is branching with a group-mean baseline in place of the greedy self-critical baseline. The third is whole-sequence GRPO plus a dense reading-accuracy reward $\lambda V$, which gives the baseline the same verifier information without locking. The fourth sets $\lambda = 0$ in our method.

Why not just…?

Why not just add a reading-accuracy reward to GRPO? This is the most serious threat. It uses the same table verifier, and it may capture most of the gain. Our bet is that shaping still re-samples readings and reasoning together, so the reasoning signal stays confounded by reading noise, and correct readings are still destroyed and must be rediscovered. The third ablation settles the question directly at a matched token budget. If shaping matches our method on accuracy and on zero-advantage rate, the locking claim fails, and we will report it that way.

Why not just tune the sampling temperature, as M-STaR does? Temperature is global. Raising it to escape saturation perturbs verified numbers as readily as uncertain reasoning, which is exactly the blind-mutation failure Driller and MuFuzz document. We run M-STaR's schedule as a baseline. The specific prediction is that it lowers the zero-advantage rate but raises the rate of corrupted verified readings in samples, while locked branching lowers the first without the second.

Why not just extract the table with a tool and let a text model reason? ChartQA found that table-only pipelines fail on questions that reference visual attributes, with its best baseline at 45.52%. GeoSQA found that even human text transcriptions of diagrams do not rescue multi-step reasoning. Our method keeps perception inside the model and uses tables only as a training-time check, so it applies at test time on charts that have no tables.

Why not just supervised fine-tuning on grounded rationales? Point-RFT's own comparison, 70.88% for SFT against 90.04% after RL, and its out-of-domain gap indicate that imitation alone leaves most of the gain unrealized. Our warm-up arm reproduces this reference point.

Decisive Checks

We will evaluate on four benchmarks. ChartQA is split by human-written and machine-generated questions. PlotQA serves as the in-domain second test. Chart101 measures elementary reading, both collective accuracy and robustness to added grid lines. MathVista and MATH-Vision test transfer to charts and diagrams without tables.

The primary result is in-domain accuracy at a matched token budget against the GRPO baseline and the shaping ablation. The mechanism checks carry equal weight. The shortcut rate, meaning correct answers built on unverified readings measured on ChartQA and PlotQA, should fall relative to GRPO. The zero-advantage fraction should stay lower late in training. Unverified locking should underperform verified locking, which would show that the validity check, not branching itself, does the work.

Chart101 collective accuracy, which Plot Twist reports below 25% for evaluated models, tests whether better reading transfers to charts never seen with tables. A null or negative result on the shaping comparison would narrow the contribution to a diagnostic of where RL credit goes, and we would report it as such.