Rewarding Only the Rollouts That Looked at the Image

When vision-language models are trained with reinforcement learning to reason step by step, the reward usually checks one thing: whether the final answer is correct. That check cannot tell a chain of reasoning that read the chart from one that guessed from linguistic habit and happened to land on the right number. We propose to test each sampled solution during training by scoring it a second time with the image blanked out. If the model finds the reasoning just as plausible without the picture, the solution is treated as ungrounded and removed from the policy update, even when its answer is correct. How aggressively to filter is not hand-set. A small bandit, a simple trial-and-error learner that tries several settings and keeps whichever pays off, picks the filtering threshold using measured gains on a held-out validation set.

The stakes are both scientific and practical. More Thinking, Less Seeing? shows that as multimodal reasoning chains lengthen, the model's attention moves from visual tokens to its own generated text, and perception accuracy degrades past a "Goldilocks" length. This is why spending more compute at test time (letting the model think longer) often hurts vision tasks when it helps text tasks. Our hypothesis is that correctness-only reinforcement learning actively trains this drift, because it rewards image-independent shortcuts whenever they happen to be right. Removing those rollouts should produce a model whose accuracy keeps rising, or at least stays flat, as its reasoning budget grows. It should also cut the number of backward passes per step, which pays for the extra scoring pass.

Core Claim

Our claim is that, in group-based reinforcement learning for vision-language reasoning, a measurable share of reward-positive rollouts carry almost no dependence on the image. Filtering them before the update will shift the accuracy-versus-reasoning-length curve upward on perception-heavy tasks without lowering visual math accuracy.

The method has three parts. We use GRPO (group relative policy optimization, which samples several answers per question and scores each against the group average instead of training a separate value network). For each rollout $y$ generated for image $x$ and question $q$, we compute a visual reliance score

$$ r(y) = \frac{1}{|y|}\sum_{t=1}^{|y|}\Big[\log p_\theta(y_t \mid y_{<t}, x, q) - \log p_\theta(y_t \mid y_{<t}, \varnothing, q)\Big], $$

where $p_\theta$ is the current policy, $y_t$ is the $t$-th token of the rollout, and $\varnothing$ denotes a blank image. We score the whole chain rather than just the answer for a specific reason. A grounded chain usually writes visual facts into the text ("the bar reaches 5"), after which the answer no longer needs the image. The evidence of grounding therefore sits in the chain's visual-fact tokens, which should be much more probable with the image than without it.

Computing $r(y)$ costs one extra teacher-forced forward pass. Teacher forcing means scoring a fixed sequence in a single parallel pass instead of generating it token by token. Because this pass is parallel, it is cheap next to the step-by-step generation that dominates reinforcement learning cost, the bottleneck described in Is Reinforcement Learning (Not) for Natural Language Processing.

Rollouts whose reliance falls below a quantile $\tau$ within the training batch are dropped from the gradient step. The value of $\tau$ is chosen by the validation-driven bandit.

Relation to Prior Work

Four papers come closest, each in a different role.

Policy Filtration for RLHF supplies the mechanism: it filters the rollout buffer before PPO updates, keeping responses where the reward model is reliable. It gains +7.9% on HumanEval pass@1 and moves Ape210K from 84.1% to 86.2%. It filters on reward-model confidence in text-only code and math. We filter on a property the reward cannot see, namely visual dependence, and in a setting where the failure is modality drift rather than reward noise.

SoTA with Less (ThinkLite-VL) selects questions by difficulty with an offline Monte Carlo tree search pass over about 70k samples before training. We select rollouts, not questions, and we select online. The two methods are orthogonal, so we test them in combination.

Omni-AutoThink masks trivial queries and penalizes superfluous thinking tokens with fixed rules, learning when to think. It controls how much the model reasons. We control whether the reasoning that gets reinforced is anchored in the image.

Two diagnostic papers motivate the work without proposing training fixes. More Thinking, Less Seeing? supplies the attention-drift finding and the RH-AUC metric, which measures area under the accuracy curve across reasoning budgets. UFO defines "evidential coupling", meaning whether intermediate cues are causally used downstream. Our reliance score is a cheap, per-rollout, training-time proxy for that coupling.

Three benchmarks establish that the gap is in reasoning over what is seen rather than in raw perception. On MATH-Vision, 42.2% of GPT-4V errors are reasoning errors after correct recognition, and chain-of-thought prompting hurt some models. GeoSQA shows that models still fail when diagrams are replaced by gold text. Plot Twist finds models doing better on high-level chart questions than on elementary readouts, which suggests linguistic shortcuts.

The validation-driven choice of $\tau$ follows the performance-driven selection tradition, such as the validation-rewarded selection policies of Learning How to Self-Learn. One constraint from that tradition matters here. Influence Functions for Efficient Data Selection in Reasoning shows that validation-driven selection works but is expensive and model-specific. We therefore keep the outer loop to a handful of arms and an occasional small evaluation, and we avoid per-example influence estimates.

Research Program

We start from Qwen2.5-VL-7B, a base model evaluated in RH-Bench, and use a GRPO codebase such as DIVA-GRPO. Training uses the open-source visual reasoning pool from which ThinkLite-VL filtered its hard subset, so the question distribution matches the strongest selection baseline.

Phase one: does the ungrounded share exist and does the score find it? Before building anything, we measure how often correct rollouts have near-zero reliance. We then test whether $r(y)$ tracks real hallucination. We hand-annotate a few hundred rollouts for invented visual facts and check the score against those labels. We also check it against the attention-to-visual-token ratio used in More Thinking, Less Seeing?.

A known risk is that hallucinated visual facts are improbable both with and without the image, which would give them a middling rather than low score. If annotation shows this, we will instead score only the tokens that mention quantities or visual entities. If neither variant separates grounded from ungrounded chains better than chance, the project stops here.

Phase two: the filtered training loop. Every $K$ steps, the bandit evaluates the policy on a held-out validation set. The set has roughly 300 perception items and 300 reasoning items, drawn from training-source splits and from VeriSciQA, and it never overlaps with the test benchmarks. Perception items are evaluated at a long reasoning budget, because that is where drift shows.

The bandit's reward is the change in the mean of perception and reasoning accuracy. Its arms are $\tau \in \{0, 0.1, 0.25, 0.4\}$. We log wall-clock time, generation time, extra scoring time and backward passes saved, so the compute claim is measured rather than asserted.

Baselines

The naive reference is the base Qwen2.5-VL-7B with no reinforcement learning, and next to it plain GRPO with correctness reward on the same pool. The plain GRPO run uses the same number of rollouts, steps and GPU-hours as our method, and is tuned over the same small learning-rate grid. It is the direct test of whether filtering adds anything. We beat it if RH-AUC rises and MathVista accuracy stays within noise of it.

The strongest existing selection method is ThinkLite-VL SoTA with Less, which reports 75.1 on MathVista for its 7B model. We reproduce its hard-subset GRPO training and run our filter both instead of it and on top of it. Its offline tree-search cost is counted in its compute budget. For drift, the strongest reported grounding result is Ocean-R1-7B at RH-AUC 0.63 on RH-Bench More Thinking, Less Seeing?.

Beating the baselines means two things. First, our filter on the ThinkLite subset must exceed 0.63 RH-AUC. Second, it must stay within one point of the reproduced ThinkLite MathVista score or improve on it. If our reproduction falls short of 75.1, we compare against the reproduced number and report the gap.

The efficiency baseline is Omni-AutoThink-style training Omni-AutoThink, with a rule-based penalty on superfluous thinking tokens. Filtration baselines are PF-PPO-style best-worst filtering on reward alone Policy Filtration for RLHF, and random dropping of the same fraction of rollouts. Random dropping matches the compute savings and isolates whether which rollouts are removed matters.

Two ablations isolate the performance-driven piece and the choice of signal. For the first, the bandit is replaced by the best fixed $\tau$ found by grid search in hindsight. For the second, $r(y)$ is replaced by the attention-ratio signal. If the fixed threshold matches the bandit, we will report the bandit as unnecessary and drop the validation-feedback claim.

Why not just…?

Why not just penalize long reasoning, since drift grows with length? This is the most serious threat. If drift is purely a function of length, a length penalty like Omni-AutoThink's would suffice. More Thinking, Less Seeing? reports a non-monotonic length effect, which hints that length is not the whole story, but that is not proof. We settle it with a length-matched comparison. We tune the length-penalty baseline until its average chain length equals ours, then compare RH-AUC. If the two tie at equal length, our filter is merely an indirect length control and we will say so.

Why not just filter by difficulty, which already works? SoTA with Less shows that removing easy questions helps, and GRPO's own group pass rate gives difficulty for free. Difficulty filtering, however, keeps every correct rollout on a hard question, including those reached by linguistic priors. The combined run tests whether the two signals add. If our filter gives no gain on top of the ThinkLite subset, rollout grounding is redundant with question difficulty.

Why not just mix perception-only questions into the reinforcement learning data? Training on questions whose answers require looking, such as MMVP-style items, might teach grounding directly and cost nothing extra. This is a plausible competitor. We run it with the perception share matched to our validation set's mix and the same compute budget. Our claim survives only if filtering beats mixing on long-budget RH-Bench perception while matching it on reasoning, or if the two combine additively.

Why not just caption the image and reason in text? GeoSQA shows that failures persist even with gold text descriptions, and UFO finds that text-only routes exploit shortcuts. We therefore do not run this as a main baseline, though we report it on MATH-Vision for reference.

Proposed Experiments

The primary evaluation is RH-Bench RH-AUC, run across the benchmark's reasoning-token budgets. It is paired with MathVista accuracy, and with MATH-Vision accuracy, where GPT-4o's reported 30.39% MATH-Vision gives an external anchor. Plotting accuracy against thinking budget makes the test-time compute claim visible. We report three seeds per configuration and GPU-hours per run.

Three decisive checks will settle the claim. First, the reliance score must predict annotated hallucination in phase one. Second, at matched chain length, the filtered model's perception-accuracy curve on RH-Bench must decline later or less steeply than those of plain GRPO and the length-penalty baseline. Third, the combined run must beat the ThinkLite reproduction on RH-AUC at no MathVista cost, with total wall-clock no worse than plain GRPO. Failing the second check would mean correctness-only reward is not what trains attention drift, and that result would be worth publishing as a negative finding.