Vision-language models trained with reinforcement learning are usually rewarded only for a correct final answer. That reward cannot tell a solution that read the diagram apart from one that guessed from the question text. It also cannot tell whether a failed attempt went wrong because the model misread the figure or because it made an arithmetic mistake. We propose a cheap fix. After each rollout, re-score the model's own reasoning a second time with the image removed. For every reasoning step, the gap between "how likely was this step given the image" and "how likely without it" says whether the image supported the step, was irrelevant to it, or contradicted it. Contradiction means the step is a likely hallucination. We use these per-step signals to redistribute the outcome reward inside each trajectory, without ever changing its sign. The same signals give an audit of how many correct answers were reached without using the image at all.
The opportunity matters because the evidence points the same way from several directions. The dominant failure on competition visual math is reasoning that goes wrong after the visual elements were perceived correctly: 42.2% "Reasoning Errors" against 31.9% "Vision Recognition Errors" in MATH-Vision. Several models score 5%+ higher with the diagram deleted (MathVerse). Outcome-only training hides answers that the agent's own record cannot support, 42.5% of correct answers in one document benchmark (AWM). The usual information-theoretic cures are too slow to put inside a reinforcement learning loop. Our bet is that the costly half of the measurement is already being paid for.
Core Claim
Mutual information (MI) is a measure of how much knowing one variable tells you about another. A pointwise version, PMI, applies the same idea to a single observed outcome. Here it compares the probability of one reasoning step with and without the image. For a reasoning trace split into steps $s_1,\dots,s_T$ (sentences or equation lines), with question $q$, image $v$, and policy $p_\theta$, we define the visual PMI of step $t$ as
$$\Delta_t = \log p_\theta(s_t \mid s_{<t}, q, v) - \log p_\theta(s_t \mid s_{<t}, q).$$
We read the sign of $\Delta_t$ in three ways. Positive $\Delta_t$ means the image made this step more likely: the step reads off visual content. Near-zero $\Delta_t$ marks image-independent steps such as algebra. Negative $\Delta_t$ means the image argues against the step, which is the signature of a perceptual hallucination.
The first term is not new work. Policy-gradient training such as Group Relative Policy Optimization (GRPO, which normalizes each rollout's reward against the other rollouts for the same prompt) already computes it. The second term needs only one teacher-forced pass over the text tokens, run in parallel with no generation and no image tokens. In typical vision-language models the image tokens make up much of the input. The ablated pass is therefore cheaper than the original prefill and far cheaper than sampling another rollout.
The claim has two parts, and both can be tested. First, step-level visual PMI used as sign-preserving credit improves accuracy on visually dependent problems relative to outcome-only RL and to a one-shot perception gate. Second, it reduces the rate of correct answers that contain no visually supported step, at a smaller wall-clock overhead than prior PMI methods.
Relation to Prior Work
The closest work is See First, Reason Later (MIRL). MIRL computes the same image-versus-no-image log-ratio, but only once, on an initial visual description. It uses the score to pick 6 of 10 candidate descriptions to continue into full reasoning, and gives the perception phase its own reward. MIRL supplies the measurement and motivates the proposal. Its 0.94 correlation between description MI and accuracy suggests the signal is real. It leaves open everything after the description, where MATH-Vision places most failures. MIRL also notes its own cost of two full passes per candidate.
We differ in three ways. We measure every reasoning step rather than one boundary. We use the score to assign credit within a trajectory rather than to filter candidates. And we reuse the policy's existing log-probabilities, so the ablated pass is the only new cost.
The credit mechanism follows TASPO. TASPO keeps the outcome reward in charge of the gradient's direction and only reshapes credit across actions, using bounded, mean-preserving weights computed from successful sibling trajectories. It reports a 10.6% average gain over GRPO on text agents. We adopt that "outcome sets the sign" constraint. Our weights come from image ablation rather than from siblings, so they are defined even when every rollout in a group fails. TASPO was never tested on visual grounding.
AWM motivates the audit. It shows that answer-only evaluation hides unsupported correct answers. Its own detector needs a frozen 14B reader model plus an LLM judge on every rollout. We test whether an intrinsic, judge-free signal recovers the same kind of diagnosis.
Demystifying Reasoning Dynamics uses a kernel dependence measure (HSIC) on text-only models. It finds that information about the answer concentrates in under 5% of tokens. That finding constrains our design: credit should be concentrated, not smeared uniformly. It is why we aggregate $\Delta_t$ at the step level and let weights peak sharply.
Two lines of work rule out heavier designs. Trained critic networks for MI estimation, as in CIBR and the instability documented by IBNorm, add a second model, a min-max game, and synchronization costs. Contrastive decoding with and without context, as in PMI-FAITH, halved generation throughput because it scored every candidate during decoding. We never decode under ablation. We only re-score a finished trace.
Research Program
Weighting rule. Each rollout receives its usual group-normalized advantage $A$ from the verifiable answer reward. Within the trajectory, each step $t$ receives a positive weight $w_t$, normalized so the weights average to one. The weights depend on the sign of $A$:
- In a successful trajectory ($A>0$), weights rise with positive $\Delta_t$. The visually supported steps are reinforced most.
- In a failed trajectory ($A<0$), weights rise with negative $\Delta_t$. Blame lands on steps the image contradicts.
- In a failed trajectory with no contradicted step, blame stays roughly uniform over the image-independent steps. This is an honest reflection that the error was logical.
Concretely, the weight is one plus a coefficient $\lambda$ times the clipped, standardized $\Delta_t$ of the relevant sign. The weights are bounded and mean-normalized, so the total reward of every trajectory is unchanged. $\Delta_t$ is computed under the rollout policy and treated as a constant, so it adds no gradient path.
Reward-hacking guard. Total reward per trajectory is fixed, so the policy gains nothing by padding traces with image restatements. Padding only spreads the same credit thinner. We nonetheless monitor trace length and the fraction of steps with positive $\Delta_t$ for drift.
Calibration of the no-image pass. A model given no image may be miscalibrated. We will compare three conditioning variants for the second term: dropping the image tokens entirely, substituting a blank image, and substituting a mean-pooled image embedding. We report how often each changes the sign of $\Delta_t$. We will also correct for token-frequency effects by length-normalizing at the step level.
Audit metric. As a by-product we define "visually unsupported correct" (VUC): a correct trajectory none of whose steps has $\Delta_t$ above a threshold. VUC is the image analogue of AWM's memory-missing-correct rate. The step labels also give a perception-versus-reasoning error decomposition for failures, which we compare against MATH-Vision's hand-labeled categories.
Baselines
Outcome-only RL. The naive reference is DAPO, an outcome-reward GRPO variant, with uniform per-token credit, run with 16 rollouts per prompt as in MIRL's DAPO-16. We train it on ViRL39K, using the same backbone, prompts, KL settings, and learning-rate sweep as our method. MIRL reports DAPO-16 at 69.76% average accuracy across its six-benchmark suite.
MIRL, the strongest competitor. MIRL is the strongest existing method because it already uses the same information signal against the same failure. Its MIRL-12 configuration reports 70.22% average accuracy on that suite with 25% fewer full trajectories, according to MIRL. We will rerun it ourselves, with its N=10/K=6 gate, on the same data and backbone. That gives same-condition numbers on MathVista, MathVerse, and MATH-Vision individually; the file does not list all six of MIRL's benchmarks.
Rollout-only comparison. Our method also runs at 12 rollouts, so the comparison against MIRL-12 isolates step-level credit from the rollout budget.
Combined method. A fourth arm applies the MIRL gate plus our step weights. This tests whether the two signals are complementary.
Fairness. Fairness is enforced in two ways. We match rollout counts. We also report accuracy at matched wall-clock GPU hours, so our extra scoring pass is charged against us.
Ablations isolating the key ingredient. Four ablations ask whether the image-ablation contrast itself matters:
- Shuffled weights keep the same weight magnitudes but scramble them across steps.
- Entropy weights replace $\Delta_t$ with per-step predictive entropy, a common credit heuristic that needs no ablation.
- Attention weights replace $\Delta_t$ with attention mass on image tokens, read for free from the original pass.
- Ablated-pass cost times the second pass alone.
Success criterion. Beating the strongest baseline means three things together:
- higher accuracy than our rerun of MIRL-12 on the vision-dominant and vision-only MathVerse variants and on MATH-Vision;
- a lower VUC rate than DAPO-16;
- wall-clock cost no worse than MIRL's.
A gain only in the average, with VUC unchanged, would count as a failure of the core claim.
Why not just…?
Why not just keep MIRL's one-shot perception gate? MIRL's gate can only reject a bad initial description. MATH-Vision attributes the largest share of errors to reasoning that follows correct perception (MATH-Vision), and a single gate cannot see those errors. MIRL's gain over DAPO-16, 0.46 points on average, is also modest. This is the main threat to the proposal. If MIRL-12 matches us on vision-dominant problems and on VUC, then step-level credit adds nothing, and the head-to-head and combined arms will show it.
Why not just sample more rollouts with outcome reward? More samples average out noise, but they do not change what is rewarded. A text-shortcut solution that reaches the right answer receives full credit no matter how many siblings are drawn. That is exactly the setting in which MathVerse finds models do better without the diagram (MathVerse). The matched-compute DAPO arm, run at 16 rollouts and at a larger budget equal to our wall-clock, will show whether VUC falls with scale alone. Our prediction is that it does not.
Why not just use an external judge or reader to grade steps? AWM's reader-judge approach works, but it calls a 14B reader and an LLM judge on every rollout (AWM). Automated graders are also fragile to formatting and to chance-agreement confounds (Actions Speak Louder than Words). We will not train with such a judge. Instead we use one, together with human annotation, to validate the VUC audit on a few hundred traces. If our cheap signal disagrees badly with the judge, the audit claim fails.
Why not just use attention to image tokens, which is free? This is a genuine threat, because SDR-MCoT gets real gains from attention reweighting with no extra pass (SDR-MCoT). Attention shows where the model looks, not whether the image changed what it wrote. A step can attend heavily to a diagram it then contradicts. The attention-weight ablation settles this directly. If it matches image-ablation credit, the cheaper signal wins and we would report that.
Proposed Experiments
Training setup. We train on ViRL39K and evaluate on MathVista, all six MathVerse variants (testmini, 788 problems), and MATH-Vision. We also use Chart101 from Plot Twist for distribution shift. The backbone is a 7B open model, Qwen2-VL-7B unless MIRL's backbone can be matched.
Shortcut check. The first decisive check is the MathVerse gap between the text-dominant and vision-only variants. Grounded credit should shrink that gap relative to DAPO-16 and MIRL-12. Accuracy on the text-dominant variant alone could rise from shortcuts as well.
Title-reliance check. The second check uses Plot Twist's title manipulations. Accuracy reached 79% when the chart title contained the answer and fell to the 30% range when the title was removed or misleading. A grounded policy should lose less when titles are altered.
Diagnostic check. The third check tests the step labels on held-out MATH-Vision failures. We sample failures, have annotators label the first erroneous step as perceptual or logical, and measure whether the most negative-$\Delta_t$ step locates perceptual errors better than the random and entropy baselines.
Cost check. The fourth check is the overhead. We report the scoring pass as a fraction of rollout wall-clock and against MIRL's two-pass gate. The proposal assumes the ablated pass is small next to generation. If it is not, the method loses its main advantage over critic- and judge-based alternatives.
Calibration check. Across the three no-image conditioning variants, we report how often the sign of $\Delta_t$ flips. If the signs are unstable, the per-step signal is not trustworthy and the weighting rule needs revision.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- AWM: Answerable Working Memory for Long-Document VQA Agents
- See First, Reason Later: Mutual Information-Guided Reinforcement Learning for Vision-Language Models
- Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
- CIBR: Cross-modal Information Bottleneck Regularization for Robust CLIP Generalization
- IBNorm: Information-Bottleneck Inspired Normalization for Representation Learning
- Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs
- Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
- Mitigating Low-Quality Reasoning in MLLMs: Self-Driven Refined Multimodal CoT with Selective Thinking and Step-wise Visual Enhancement
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details