Counterfactual Group Rewards for Vision-Language Models That Read the Picture

Vision-language models often answer chart and diagram questions from the text around the image rather than from the image itself. On simple chart-reading questions, accuracy reached 79% when the chart title contained the answer and fell to about 30% when the title was removed or made misleading (Plot Twist). The same models also flip answers when a prompt is harmlessly reworded (ProSA). We propose a training reward that attacks both problems at once, and it needs no hand-written reward rules or judge model. For each training question we build a small family of variants whose correct answers are known by construction. Some variants change only the wording, which should not change the answer. Others change the plotted data, which should change the answer in a way a program can compute. The model earns reward on a question only to the extent that it answers the whole family correctly.

The two-sentence version is this. Reward a model for giving the same answer when only the wording changes and a different, correct answer when the picture changes. This teaches it to track visual evidence instead of text shortcuts, and the key open question is whether that habit transfers to image domains where no such variants can be generated.

Core Claim

Current multimodal reinforcement learning with verifiable rewards scores each sampled answer against a single gold label. This pays off "lucky guesses" and shortcut answers whenever the shortcut happens to agree with the label. Two lines of work patch this. Perception-R1 adds a reward for listing visual facts, checked by a 32B judge model; it found that a 7B judge was gamed by hallucinated descriptions. Self-Consistency Sampling scales rewards by answer stability under reasoning truncation and Gaussian image noise. Both reward invariance or description. Neither rewards equivariance, meaning that the answer moves correctly when the evidence moves. That is exactly the property the title-flip result in Plot Twist shows is missing.

Our claim is that a reward coupling invariance to wording with correct change under visual edits removes the text shortcut more effectively than rewarding the same variants one at a time. We further claim that the resulting behavior carries over to held-out visual domains (competition geometry, scientific figures) and lowers prompt sensitivity there.

Formally, for seed item $x$ let $S(x)$ be its set of variants and $c_j \in \{0,1\}$ the correctness of a policy sample on variant $j$. The reward for a rollout on variant $i$ is

$$r_i = c_i \cdot \frac{1}{|S(x)|}\sum_{j \in S(x)} c_j ,$$

so a correct answer is worth little if the model fails the siblings that differ only in the evidence that should matter. This is a group-relative reward in the sense of Group Relative Policy Optimization (GRPO, which normalizes rewards within a group of samples for the same prompt). Here the group spans counterfactual inputs, not just resamples.

Research Program

The training domain is programmatically rendered charts and function plots. This is where Synthesize Step-by-Step already showed that data-to-image pipelines give exact answers. Each seed is a data table, a rendering script, and a question with an executable answer. From each seed we generate three kinds of variants.

Text variants include paraphrased instructions, reordered or relabeled options (the perturbations shown to reshuffle leaderboards in When Benchmarks are Targets), and titles that are neutral, answer-revealing, or misleading. These share the gold answer.

Style variants re-render the same data with different libraries, grid lines, and markers, which Plot Twist found to break reading accuracy. These also share the gold answer.

Evidence variants edit the underlying data and recompute the answer by executing the question program. We deliberately pair an evidence edit with a title that still states the old answer, so the text shortcut is directly punished.

This is how the proposal handles the manual-reward-engineering direction. The only human input is the renderer and the question templates. There is no per-task reward formula, no perception judge, and no trained verifier of the kind StructVRM needs. Process reward models and multimodal judges prefer fluent text over correct content (Reward Under Attack; Perceptual Judgment Bias), so avoiding a learned grader removes a known attack surface rather than just saving compute.

The cross-domain direction supplies the central test. The literature on transferable agent memory, such as Learning How to Remember, argues that abstract skills transfer while instance-tied tactics do not. We ask the analogous question for perception. Does "check the plotted values, ignore the title" become a general habit that shows up on MATH-Vision geometry and VeriSciQA scientific figures, where we cannot generate counterfactuals at all?

The prompt-sensitivity direction supplies both a measured outcome and a mechanism hypothesis. ProSA links sensitivity to low decoding confidence, and SENSEL finds sensitivity correlates negatively with accuracy (up to −0.40). We therefore predict that a model trained to be invariant under the joint reward will show lower sensitivity scores on held-out domains, and that its remaining sensitivity will flag its errors.

One cited finding constrains our expectations. MATH-Vision attributes 42.2% of GPT-4V errors to reasoning and 31.9% to recognition. GeoSQA finds models fail at similar rates even when diagrams are replaced by text. Evidence-tracking cannot fix pure deduction errors. We therefore predict gains concentrated on perception-heavy MATH-Vision subjects and small gains on subjects where reasoning dominates. If gains instead appear uniformly, our mechanism story is wrong.

Baselines

All trained methods start from Qwen2.5-VL-7B and use GRPO with the same number of policy rollouts and gradient steps. Counterfactual siblings count against the rollout budget, so our method sees fewer distinct seeds, not more compute. Each method receives the same hyperparameter search budget.

The do-nothing reference is the base Qwen2.5-VL-7B, which Perception-R1 reports at 68.1% on MathVista.

The first trained baseline is standard outcome-reward GRPO on the seed charts only. This is the default practice that Perception-R1 found produces no significant perception improvement.

The ablation that isolates our key ingredient is pooled augmentation. It uses GRPO on exactly the same variant set, but each variant is rewarded independently against its own label, so the group coupling is removed. This is the most dangerous competitor, because it sees the same shortcut-breaking examples.

We also compare to Self-Consistency Sampling reimplemented on our data. It is the closest existing reward-shaping method (reported +7.7 points averaged over six benchmarks), and the comparison checks whether invariance-only consistency is enough.

The strongest published method is Perception-R1, reported at 74.2% on MathVista with under 1,500 samples. We run it with its 32B judge on our chart seeds, and also with its original geometry data, reporting its compute cost alongside.

Beating a baseline means the following. On in-domain probes, we must exceed pooled augmentation on the misleading-title gap and the all-variants-correct rate. On held-out benchmarks, we must match or exceed Perception-R1 on MathVista (74.2%) without a judge. For MATH-Vision, no reported result for these trained 7B baselines was found on Lacuna, so we run all baselines ourselves.

Why not just…?

Why not just train on the augmented variants with ordinary per-sample rewards? This is a real threat. Under per-sample rewards a shortcut answer is already wrong on misleading-title variants, so the model may learn the same thing. Our argument is that independent rewards still pay the shortcut fully on the majority of variants where text and image agree. The joint reward withholds most of that credit unless the model also handles the conflicting siblings. The pooled-augmentation ablation settles it directly. If it matches us on the title-flip gap and on held-out transfer, we will report that coupling is unnecessary and that the contribution reduces to the data generator.

Why not just use a perception reward with a strong judge, as Perception-R1 does? Perception-R1 shows it works on geometry. However, it requires Gemini-extracted annotations and a 32B judge, and it was gamed at 7B. It also never tests whether the model's answer depends on the image when text disagrees. Describing visual facts is not the same as using them. We compare on MathVista and on our misleading-title probe, where a model can list correct facts and still answer from the title.

Why not just ensemble over prompt paraphrases at test time? This is the SENSEL/ProSA route, and few-shot anchoring is ProSA's largest single robustness gain. But it multiplies inference cost, and it cannot fix answers that are stably wrong. A model that always reads the title is perfectly consistent. We include majority voting over the same text variants at inference as a comparison and report accuracy per forward pass.

Why not just fine-tune on rendered charts with executable rationales? Synthesize Step-by-Step raised ChartQA human-split accuracy from 38.2% to 53.9% this way. However, VeriSciQA finds supervised gains on held-out benchmarks small (+2.05% average), and supervised learning does not penalize shortcut reliance. We add a supervised baseline on identical seeds and measure the title-flip gap.

Proposed Experiments and Decisive Checks

In-domain, we build a Chart101-style probe from held-out seeds and report three quantities: accuracy, the gap between answer-revealing and misleading titles (Plot Twist reports about 79% versus about 30% for off-the-shelf models), and the fraction of seed groups answered entirely correctly.

For transfer, we report MathVista, MATH-Vision broken down by subject, ChartQA, and the VeriSciQA resource (art_d424ac063abc47f7bb69ba8a83226a43). None of these are used in training. For prompt sensitivity, we apply ProSA's twelve templates to MathVista and MATH-Vision and compute its PromptSensiScore, plus the SciAux resource (art_e5ba7d32c8e7450596d2e9442ee21a24) for misleading-context robustness.

The decisive checks are three. First, the joint reward must beat pooled augmentation on held-out domains, not just on charts; this confirms that coupling, not data, drives transfer. Second, MATH-Vision gains should concentrate in perception-heavy subjects, consistent with the error decomposition above. Third, the sensitivity score should drop on held-out domains, and residual sensitivity should predict errors at least as well as it does for the base model. If the first check fails, the proposal collapses to data augmentation, and we will say so.