Rewarding Perception That Tracks the Image: Counterfactual Twins for Multimodal Reasoning

Vision-language models often answer chart and diagram questions correctly for the wrong reasons. They read the answer off a chart title, guess from wording in the question, or latch onto a background cue, and then write a plausible step-by-step explanation that never mentions the shortcut. Recent training methods try to fix multimodal reasoning by rewarding the model's description of the image whenever that description is enough for a separate text-only model to solve the problem. This proposal starts from a simple observation: a description that copies the shortcut is also "enough." A transcribed chart title lets the text-only model answer correctly, so the perception step gets rewarded for learning exactly the wrong habit.

We propose to reward the model's description of an image only if it also behaves correctly on a counterfactual twin of that image. A twin is a programmatically re-rendered version of the same chart or diagram in which either the answer-relevant data change while the shortcut stays fixed, or the shortcut changes while the answer stays fixed. A description earns credit only when a frozen text-only reasoner reaches the right answer on the original and on its twins. Put plainly, the model must describe the parts of the image that actually determine the answer, and changing those parts must change what it says. Because the check is an intervention on the underlying data rather than an occlusion of one input, it also catches shortcuts that live in the combination of question and image. Those are shortcuts that hiding one input at a time provably misses.

Core Claim

Sufficiency-based perception rewards, as in Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning (MoCA), cannot tell a faithful description from one that transcribes a spurious cue. Requiring descriptions to stay sufficient across paired interventions on the image's generating program can tell them apart. On charts, where such interventions are cheap and exact, this should sharply shrink the shortcut gaps documented by Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details (Chart101) without lowering standard chart accuracy. The improvement should come from the paired reward, not merely from training on more varied charts.

How This Relates to Prior Work

Two papers motivate the problem. Chart101 shows that leading models reach accuracy "as high as 79%" when titles carry answer cues and fall to "the 30% range" when titles are neutralized. It also shows that cosmetic edits such as grid lines move PaLI-3 from 29.5% to 21.3%. The paper offers diagnosis only and no training fix. Language Models Don't Always Say What They Think shows that chain-of-thought explanations (step-by-step written reasoning before the answer) omit the biasing feature in 425 of 426 cases. This is why we never trust the reasoning text's own account of what it used; we test it by intervention instead.

Two papers constrain the design. Misleading Failures of Partial-input Baselines proves that hiding one input at a time misses relational artifacts. Its hypothesis-plus-one-word model solves 15% of the SNLI "hard" subset that hypothesis-only probes certified as clean. This rules out the design of simply adding an image-blind or question-blind bias detector to MoCA. Seeing What's Not There (SpurLens) finds that spurious visual bias sits in the vision encoder's embeddings and that chain-of-thought and guiding prompts fail to close the gap. So our method must update weights, and we test whether the vision encoder itself needs to move.

Three papers supply method. MoCA supplies the split output format and the reward machinery. Its format puts visual facts in <recognition> blocks and deduction in <think> blocks, and a frozen "blindfolded" Qwen2.5-14B that sees only the recognition text decides whether perception gets credit. We keep all of this and change only the condition under which perception is rewarded. Reasoning-Aligned Perception Decoupling (RAPID) trains a perceiver with rewards from a frozen external reasoner. It notes that the reasoner stays exposed to textual shortcuts and hallucinated artifacts passed along by the perceiver, which is precisely what our twins are meant to filter. Answer-Consistent Chain-of-thought RL (ACRE) is the closest relative in spirit. It also rewards consistency under a perturbation, but its perturbation is shuffling the order of multiple-choice options, which only targets position bias. Our perturbations act on image content, and they also apply to open-ended numeric answers, where ACRE has no mechanism.

Grounded Chain-of-Thought asks models to emit bounding boxes before answering. Bounding boxes cannot represent axis values or line trends, and that paper's finding that 72B models ground worse than 7B ones is a second reason not to expect scale alone to fix this.

Research Program

The first component is a twin generator for charts. We regenerate charts from their plotting programs, using synthetic Matplotlib and JavaScript-style charts in the spirit of Chart101, plus ChartQA charts whose underlying data tables can be re-plotted. For each training question we produce two kinds of twin. An answer-changing twin edits the data so that the correct answer moves, while the title, colors, layout, and question text stay identical. An answer-preserving twin keeps the data but changes the title to neutral or misleading text and applies cosmetic edits (grid lines, marker shape, style), so the correct answer stays the same. Some answer-changing twins are built to break joint cues on purpose. For example, if the question names "the orange line," we permute which series is orange, so the right answer depends on reading the color in the image rather than any fixed association between question wording and position.

The second component is the reward. Let $p(x)$ be the policy's recognition text for image $x$, let $R$ be the frozen text-only reasoner, and let $\mathcal{T}(x)$ be the set of twins of $x$, each with correct answer $y_{x'}$. The perception reward is

$$ r_{\text{perc}}(x) = \mathbf{1}\left[R(p(x)) = y_x\right] \cdot \frac{1}{|\mathcal{T}(x)|}\sum_{x' \in \mathcal{T}(x)} \mathbf{1}\left[R(p(x')) = y_{x'}\right] $$

This reward is high only when the description is sufficient on the original image and stays sufficient as the image changes. A copied title fails the answer-changing twins. A description driven by the joint question-image cue fails the color-permuted twins. We plug this term into MoCA's credit routing unchanged. That routing protects perception tokens when perception is judged sufficient but the deduction is wrong. Within each group of sampled answers in the RL algorithm, twins are rolled out with the same policy, so the extra cost is a second generation pass, comparable to ACRE's reported 24% overhead.

The third component is analysis. We probe whether the improvement lives in the language model or the vision encoder by training two versions: one with the encoder frozen and one with low-rank adapters on the encoder. SpurLens predicts that only the second will close cosmetic and background gaps. We will check that prediction directly.

Baselines

The do-nothing reference point is Qwen2.5-VL-7B prompted with zero-shot chain-of-thought and no further training. It shows how large the title and cosmetic gaps are before any intervention. SpurLens and the MATH-Vision paper both suggest that prompting alone will barely move these gaps.

The first trained baseline is plain outcome-reward GRPO on the same base model and data, where GRPO is the group-relative policy optimization algorithm used by MoCA, RAPID, and ACRE. This is the standard recipe the others improve on.

The strongest competitor is MoCA itself, retrained by us on the same base model, the same 14B blindfolded reasoner, and the same chart training set. That way the only difference from our method is the twin condition in the perception reward. MoCA is the right foil because it already separates seeing from thinking and already uses a blind proxy, so any advantage we show isolates the specific failure we claim. For reference, MoCA's VL-Scaler-7B reports 86.6 on V* and 45.1 on DUDE. No reported result found on Lacuna for MoCA, RAPID, or ACRE on Chart101.

We also run RAPID's perceiver-reasoner pipeline with the reasoner fixed to the same size. RAPID reports 53.2% average accuracy on its seven-benchmark suite for Qwen2.5-VL-7B with a Qwen3-8B reasoner. We also run ACRE's option-shuffling reward on the multiple-choice subset. ACRE reports option-shuffle consistency of 74.6% versus 49.8% for GRPO on MMVU.

The decisive ablation is twin augmentation with outcome reward only. This uses exactly the same original-plus-twin training images as our method, each treated as an independent example with the standard answer reward and MoCA's sufficiency reward, but with no pairing in the reward. A second ablation keeps the pairing but uses only answer-preserving twins. This tests whether answer-changing interventions, the ones aimed at relational shortcuts, carry the effect.

Every arm gets the same number of RL steps and the same count of rendered images, so the twin arms do not receive extra data. Every arm also gets the same hyperparameter search budget, selected on a held-out validation split of synthetic charts that excludes the twin types used in testing. Beating MoCA means a smaller title gap and lower cosmetic flip rate on Chart101, with ChartQA accuracy within one point of MoCA's. Beating the augmentation ablation means the same, at equal data.

Why not just…?

Why not just train on counterfactual charts with an ordinary reward? This is the real threat, because data augmentation alone is a well-understood fix for spurious correlations. If showing the model neutral-title and re-plotted charts as separate examples closes the gaps, the paired reward is unnecessary. The augmentation ablation above settles this directly. Our hypothesis for why pairing should matter is that, with independent examples, the model can still be rewarded on the large majority of charts where the title happens to agree with the data. The paired reward removes credit for that strategy on every example. If the ablation matches our method, we will report augmentation as the finding and drop the paired-reward claim.

Why not just extend MoCA with a question-only or image-blind bias detector, the classic partial-input recipe? Feng et al. in Misleading Failures of Partial-input Baselines show that such detectors give a false clean bill of health whenever the shortcut is spread across inputs. MMMU-Pro's text-only filtering, followed by drops of 16.8% to 26.9% under stricter evaluation, shows that one-input filtering leaves much undiscovered. We include the color-permuted twins and report results separately on that relational subset, where a partial-input detector should help least.

Why not just neutralize titles at test time, or prompt the model to ignore them? Titles are one shortcut among many. Chart101's grid-line and marker-shape swings are not about text at all. Real charts cannot be reliably stripped of every cue at inference. SpurLens reports that guiding prompts offer only marginal improvements and fail to close the spurious gap. We still run title neutralization at inference as a cheap reference on Chart101. We expect it to fix the title gap alone and leave the cosmetic flip rate unchanged.

Why not just use option shuffling as in ACRE? Shuffling changes where the right option sits, not what the image says, so it is blind to title and layout shortcuts and does not apply to numeric answers. Running ACRE as a baseline shows whether consistency under option order carries over to consistency under image interventions. We expect it does not.

Proposed Experiments

The primary test bed is Chart101, which offers elementary unit questions scored within a 5% tolerance. On it we report three numbers: the title gap (accuracy with answer-bearing titles minus accuracy with neutralized titles), the cosmetic flip rate (the fraction of answers that change under grid-line, marker, or style edits), and the intervention tracking rate (the fraction of held-out answer-changing twins on which the answer moves to the new correct value). Held-out twins use edit types never seen in training, such as unseen color palettes and fonts, so we do not simply reward memorizing our generator.

In-distribution accuracy is checked on ChartQA, split into its human-written and machine-generated subsets, with the original paper's best reported accuracy of 45.52% as a historical anchor. To check that chart-trained perception does not cost general reasoning, which would be the "seesaw effect" MoCA describes, we evaluate on MathVista, MATH-Vision, and MMMU-Pro. MMMU-Pro is included in both standard and vision-only settings, since it was built to remove text-solvable shortcuts.

As a faithfulness check that does not trust self-reports, we feed recognition text produced on an original image to the blind reasoner alongside the twin's question. If that text still yields the original answer on an answer-changing twin, the description encoded the shortcut. We report how often this happens for each method.

The outcome that would most strongly support the proposal is a specific joint result on Chart101. Our method roughly halves the title gap relative to MoCA, and tracks unseen interventions substantially better than the augmentation-only ablation at equal data, while matching MoCA on ChartQA and MathVista. A result where augmentation alone matches us, or where gains appear only with the encoder unfrozen, would each redirect the claim. The first would mean the twins matter and the pairing does not. The second would mean the shortcut sits where SpurLens says it does, and the fix has to reach the vision encoder.