Multimodal models often get a diagram question right for the wrong reason. They write a step-by-step explanation that mentions the figure, but the final answer actually came from wording in the question, a chart title, or a prior guess. Such a model is fragile: rephrasing the instruction or adding grid lines to a chart can flip its answer, because the answer was never anchored to the visual evidence it claims to use. This note proposes a simple training signal that targets this failure directly. A sampled solution earns reward 1 only if two things hold. Its final answer is correct, and corrupting the visual reading it reports (for example, changing "the bar for 2019 is 42" to "the bar for 2019 is 57") and letting the model finish the solution changes that answer. Every other solution earns 0. The model is paid for correct answers that pass through its own stated perception.
The bet is that this one pass/fail check, used inside reinforcement learning (RL), does two jobs at once. It reduces post-hoc rationalization, meaning explanations written after the answer was already decided. It also reduces prompt sensitivity, meaning answer flips under meaning-preserving rewordings. The model is never shown a reworded prompt during training. If that holds, it would mean that much of the instructional brittleness seen in multimodal RL is a symptom of answers that bypass the reasoning trace, not a separate formatting problem that needs its own fix.
Core Claim
Two text-only findings sit side by side in the literature without being connected. Prompt sensitivity marks examples the model does not really understand: On the Relation between Sensitivity and Accuracy in In-Context Learning finds sensitivity negatively correlated with correctness, with Pearson $r$ reaching $-0.40$. Separately, the reasoning trace and the answer are often computed in parallel from the prompt rather than in sequence. How Likely Do LLMs with CoT Mimic Human Reasoning? shows this with causal analysis, and Unveiling Confirmation Bias in Chain-of-Thought Reasoning shows models overriding steps that contradict their initial hunch.
The multimodal picture suggests a link between the two. MATHVERSE finds that removing the diagram can raise accuracy by 5% or more. Plot Twist finds that accuracy falls from 79% to about 30% when an answer-bearing chart title is removed. If the answer route skips the visual reading, any change to the non-visual context (a title, a template, a phrasing) moves the answer.
Our claim has two parts, each testable on its own. First, in current vision-language models, examples whose answers are insensitive to corruption of the stated visual reading are disproportionately the examples that flip under prompt perturbation. Second, a binary reward that requires answer correctness and causal dependence on the stated reading lowers prompt sensitivity on perturbation types never seen in training. We expect it to match or beat methods that train on perturbed prompts explicitly, while also raising accuracy on vision-heavy problem variants.
Relation to Prior Work
The closest work falls into three groups, and each plays a different role here.
The first group supplies the method. FRIT replaces a reasoning step with a style-matched distractor and checks whether the answer flips. It uses this to build faithful and unfaithful trace pairs for offline preference training on GSM8K and StrategyQA. We reuse its intervention test but change three things. We apply it to the step that reports a visual reading. We run it online, so the reward is computed on the model's current samples rather than on a fixed dataset. And we use it as the reward itself rather than to curate pairs. FRIT also warns that RL updates can create new shortcuts that erode faithfulness. Online checking is our direct answer to that warning, because the check is recomputed on fresh samples throughout training.
The second group motivates the binary design. Train for Truth, Keep the Skills shows that a pass/fail factuality reward reduces hallucination while keeping chat quality (a −1.4% AlpacaEval change). A continuous factuality score instead teaches stylistic gaming (a 22.8% drop). SketchThinker-R1 finds that binary judgments steer multimodal reasoning more reliably than dense scores, and Unified Reinforcement and Imitation Learning for Vision-Language Models reports similar stabilization for vision-language policies. URSA is the counterpoint. It argues that multimodal math needs a continuous process reward model, a learned scorer of each intermediate step, and it had to invent a relative "drop-moment" signal because absolute step scores were gamed. We take URSA's gaming problem as evidence for discretizing. URSA's step scores also never test whether a step causally matters.
The third group constrains the design. Improving Generalization Robustness of Multimodal RLVR (PIRL) shows that RL with automatically checkable rewards over-fits to prompt templates. It also shows that plain 0/1 rewards conflate formatting failures with reasoning failures. Its remedy combines a three-valued reward, separate normalization of the format and correctness signals, and adversarial perturbation of instruction embeddings. We take its entanglement warning seriously: answer extraction in our reward is deliberately lenient, so format errors do not masquerade as faithfulness failures. Our hypothesis differs from PIRL's, though. PIRL treats brittleness as template overfitting to be trained away. We test whether brittleness is downstream of trace bypass.
TIGER mines pairs where a problem is solved as text but fails as an image, and trains on them to push visual tokens into the model's internal reasoning pathway. That improves outcomes but never asks whether the written trace carries the answer. Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation supplies a negative result we rely on: instructing a model not to anchor on the answer increases anchoring. That rules out a prompting-only fix.
Research Program
The work has three stages.
Stage 1: diagnostic. We use Qwen2.5-VL-7B and Qwen3-VL-8B, the models used by PIRL and TIGER. We prompt them to write solutions in which visual readings appear as explicit, parseable claims: values, lengths, angles, labels, axis ranges. For each correct solution we perform the intervention. We replace one reported reading with a plausible distractor of the same type, truncate the trace there, and let the model continue greedily. An answer is reading-dependent if at least one of $k=3$ distractors changes it. Using several distractors guards against cases where the answer legitimately does not change.
Separately, each item gets a flip rate: the fraction of eight meaning-preserving prompt variants under which the answer changes. Instruction paraphrases and option reorderings follow the families documented in When Benchmarks are Targets and The Butterfly Effect of Altering Prompts. Visual-context variants follow Chart101: title removal or misleading titles, and added grid lines.
The test is whether reading-independent correct answers have higher flip rates. We control for model confidence, since ProSA links sensitivity to flat output distributions. The question is whether bypass predicts brittleness beyond what confidence already explains.
Stage 2: training. We run group-relative policy optimization (GRPO), an RL method that scores each sampled solution against the others drawn for the same question. Writing $a(y)$ for the extracted answer of trace $y$, $a^\*$ for the gold answer, and $\tilde y_j$ for the continuation after the $j$-th distractor edit, the reward is
$$r(y) = \mathbb{1}[a(y)=a^\*]\cdot \mathbb{1}\big[\exists\, j\le k:\ a(\tilde y_j)\ne a(y)\big].$$
Traces with no parseable visual reading score 0 on any problem whose gold solution requires one. The second check runs only on correct rollouts, so the extra cost is at most $k$ short continuations for each correct sample. This addresses the overhead problem raised against online causal auditing.
Training data are rendered and chart problems with verifiable answers. We use the image-rendered OpenR1-Math problems from TIGER and SketchRL-style multimodal questions. Crucially, training contains no paraphrased or perturbed prompts.
Stage 3: evaluation. Evaluation uses held-out perturbation families and a faithfulness measure built from a different intervention than the one in the reward. That measure edits the image itself on synthetic charts, where the correct new answer is computable. It asks whether the model's answer moves correctly when the pixels change. Using the training intervention to measure faithfulness would be circular, so we do not.
Baselines
The naive reference is outcome-only GRPO with a 0/1 correctness reward on the same data, model, rollout count, and step budget. This is the setting PIRL reports as brittle: about a 3% average drop under template stress, and −2.3% on VQA-RAD. We also report the untrained model with standard step-by-step prompting.
The strongest competitor on prompt robustness is PIRL itself, from Improving Generalization Robustness of Multimodal RLVR. It reports an average accuracy drop of at most 1% under template stress and a +0.8% change on VQA-RAD. It is the strongest because it attacks exactly our target metric with explicit invariance training. We reimplement it on our training set with the same base model and GPU-hour budget, and tune its consistency weight over the same number of trials we spend on our $k$ and distractor settings. Beating it means a smaller accuracy drop and lower flip rate on perturbation families that neither method trained on. Matching it on PIRL-style template stress while winning on visual-context perturbations would also count, since PIRL never sees those.
The strongest competitor on accuracy is URSA's process-supervised RL. It reports 31.5% on MATH-Vision for URSA-8B-PS-GRPO, against 30.4% for GPT-4o, in Unlocking Multimodal Mathematical Reasoning via Process Reward Model. We compare against the released number and, where compute allows, against PS-GRPO run from our base model. Our goal is not to beat it on MATH-Vision; it is to stay within about a point of it while being clearly more prompt-robust. For TIGER we use its reported ~15% MathVerse gain on Qwen3-VL-8B as a reference. No reported MathVerse number by version was found on Lacuna for URSA or TIGER.
Three ablations isolate the key ingredients. A continuous variant replaces the dependence indicator with the fraction of the $k$ distractors that change the answer. This tests whether binarization matters, as the factuality results predict. An offline FRIT-style variant builds faithful and unfaithful pairs from base-model samples using our intervention and trains with direct preference optimization (DPO). This tests whether online checking matters. A "mention-only" variant rewards correct answers that merely contain a parseable reading, with no intervention. This tests whether the causal check, rather than the format of explicit readings, does the work.
Why not just…?
Why not just train on paraphrased prompts? This is the most serious threat. Outcome GRPO with heavy prompt augmentation may well match us on any perturbation family it was trained on. We will run it with the same eight-variant augmentation used in evaluation, for a fair upper bound. The decisive comparison is on held-out families, especially visual-context changes like title edits and grid lines, which no instruction augmentation covers. If augmentation matches us there too, the "bypass causes brittleness" story is wrong, and we will report it that way.
Why not just port FRIT directly? FRIT's offline pairs come from the starting model. Its own discussion warns that RL-style updates create new shortcuts, so a fixed dataset of faithful traces can go stale. The offline-DPO ablation measures exactly this. We expect faithfulness under image edits to decay over training for the offline variant and not for the online one.
Why not just use a continuous process reward model? Train for Truth, Keep the Skills shows continuous scores invite gaming, and URSA itself needed a relative signal to stop length bias. A conventional process reward model also scores whether a step looks correct, not whether the answer depends on it. A step can be correct and decorative, the failure mode documented in Can Aha Moments Be Fake?. The continuous-dependence ablation and the URSA comparison test both points.
Why not just tell the model to reason from the diagram? Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation finds suppression instructions increase answer anchoring. MATH-Vision finds step-by-step prompting can lower multimodal accuracy. We include an instructed-prompting arm in Stage 1 as a check.
Proposed Experiments and Decisive Checks
Evaluation covers four groups of benchmarks. MathVerse is reported separately for its text-dominant and vision-only versions. MATH-Vision and MathVista measure general accuracy. Chart101 elementary questions, run with and without its title and grid-line manipulations, measure visual-context robustness. A template-stress protocol mirroring PIRL's covers MathVista and VQA-RAD.
The metrics are four. Accuracy is the first. The per-item flip rate is computed over held-out perturbation families. The vision-only minus text-dominant accuracy gap on MathVerse is a direct read on the "Text-Only Paradox". Image-edit faithfulness on synthetic charts asks whether the answer tracks a pixel change correctly. We also report the false-negative rate of the dependence check on a hand-audited sample of 300 traces, because Train for Truth, Keep the Skills saw 16% false negatives in its own binary verifier and we need to know ours.
Three results would decide the question. First, Stage 1 must show that reading-independent correct answers have reliably higher flip rates than reading-dependent ones after controlling for confidence. If they do not, the link hypothesis fails, and the training stage becomes only a faithfulness method. Second, the trained model must reduce held-out flip rate relative to outcome GRPO, and match or beat PIRL and paraphrase augmentation on visual-context perturbations, with no perturbed prompts in training. Third, the binary reward must beat both its continuous variant and the mention-only variant on image-edit faithfulness, with MATH-Vision accuracy within about one point of URSA's reported 31.5%. Passing all three would show that rewarding answers that pass through the model's own perception is a single lever on both rationalization and brittleness.
References
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- How Likely Do LLMs with CoT Mimic Human Reasoning?
- Unveiling Confirmation Bias in Chain-of-Thought Reasoning
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
- SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models
- Unified Reinforcement and Imitation Learning for Vision-Language Models
- Unlocking Multimodal Mathematical Reasoning via Process Reward Model
- Improving Generalization Robustness of Multimodal RLVR
- TIGER: Bridging the Multimodal Reasoning-Access Gap via Modality Counterfactuals
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset