Recycling Solved Rollouts as Curriculum Visual Negatives for Multimodal RL

When a vision-language model is trained with reinforcement learning (RL) on math problems, the reward only checks whether the final answer is right. That reward cannot tell whether the model looked at the diagram. It is also expensive, because every training signal requires generating a full reasoning chain token by token. A large share of that generation is wasted. When every sampled answer to a prompt is correct, the group-relative update gives each answer zero advantage, so the rollouts contribute nothing. This note proposes putting those "solved" rollouts back to work.

For each correct reasoning chain, we re-score it in one parallel forward pass, conditioned on a negative image: a different or subtly edited diagram under which that reasoning should no longer hold. We then penalize the model if the reasoning stays just as likely without the real picture. The negatives follow a curriculum, starting with a blank image, then an unrelated image, then a same-topic diagram, then a minimally edited diagram. The curriculum advances online according to how well the model currently separates true from negative images, not according to a fixed epoch schedule.

Put in two sentences: RL for visual reasoning throws away the rollouts it already solved and never checks that answers depend on the image. We turn those rollouts into cheap, progressively harder "wrong picture" tests, costing one forward pass each instead of a new rollout, to make the model actually use vision. We expect the payoff to be largest in unified multimodal transformers, meaning single-stream models where image patches and text tokens share one self-attention stack from the first layer. In those models image tokens sit inside the policy's own generation cache, so rollouts are costliest.

Core Claim

At a fixed compute budget, adding a teacher-forced contrastive term on curriculum-scheduled negative images to group-based RL will do two things on vision-dependent benchmark splits. It will improve accuracy more than spending the same compute on additional rollouts. It will also shrink the gap between vision-dependent and text-dominant performance, the "Clever Hans" gap in which a model answers from the wording of the question rather than the picture.

The mechanism is a likelihood margin. Let $y$ be a correct rollout of length $|y|$ for question $x$ and true image $I$, let $I^-_k$ be a negative image at difficulty tier $k$, and let $\pi$ be the policy. Define

$$m_k = \frac{1}{|y|}\left[\log \pi(y \mid x, I) - \log \pi(y \mid x, I^-_k)\right], \qquad \mathcal{L}_{\text{neg}} = \max(0,\ \delta - m_k),$$

where $\delta$ is a target margin. The term is added to the RL loss with weight $\lambda$.

The curriculum picks tier $k$ per prompt so that the hinge is active about half the time. This borrows the "learning boundary" logic that Single-stream Policy Optimization applies to prompt success probability, and applies it instead to image discrimination.

The cost argument is simple. A rollout decodes $|y|$ tokens sequentially. The negative check is a single parallel prefill over image, question, and $y$, and needs no reward call. This is the same efficiency asymmetry between parallel teacher forcing and sequential generation that Teacher Forcing Recovers Reward Functions for Text Generation exploits in text.

How This Differs From Existing Work

Three papers motivate the gap by showing where visual RL fails.

  • MATHVERSE builds problem variants ranging from text-dominant to vision-only. It finds that some models score 5%+ higher when the diagram is removed.
  • VL-Cogito runs a curriculum of group-relative RL (GRPO, which replaces the learned value network with the average score of several sampled answers to the same prompt) on Qwen2.5-VL. It weights prompts by rollout accuracy but uses only answer-correctness rewards, so nothing in its objective asks whether the image mattered.
  • Infi-MMR stages training as text first, then captioned images, then raw images. This forces grounding only through hand-set phase boundaries.

Our proposal keeps their RL loop and adds an explicit, per-sample test of image dependence that they lack.

Two papers supply the negative-sampling method we adapt, and one of them is the closest competitor.

  • Aligning by Misaligning (BACL) schedules negatives from easy to near-boundary with a logistic, epoch-based schedule for dual-encoder contrastive pretraining. We move that idea into generative RL and replace the epoch schedule with a margin-driven one.
  • Semantic Curriculum Preference Optimization (SCPO) is the closest prior work. It uses an easy-to-hard curriculum of minimally different pairs and a symmetry loss comparing outputs with and without the image, inside offline preference optimization (DPO) on pre-sorted data.

We differ from SCPO in three ways. Our positives are the policy's own fresh, verified-correct rollouts rather than a fixed offline dataset. Our negatives are images, not texts, and their difficulty adapts online. And the signal is produced precisely where the RL loop would otherwise waste compute.

MISP-DPO constrains the design. It finds that three semantically distinct negatives beat larger sets and that more than five degrade performance. We therefore cap negatives per rollout at a small number and pick them from distinct tiers.

Adapting LLaMA Decoder to Vision Transformer shows that causal decoders can process images well given a post-sequence class token and a bidirectional-to-causal soft mask. It is relevant because a unified decoder is where our cost claim is strongest. We do not claim its techniques interact with our loss.

Research Program

Negative bank. We build a four-tier bank for each training problem:

  • Tier 0 is a blank image.
  • Tier 1 is an image from a different source.
  • Tier 2 is a diagram from the same subject and source.
  • Tier 3 is a minimally edited version of the true diagram, for example one numeric label changed or one segment moved.

Tier 3 is where false negatives become a risk, because an edit might leave the answer unchanged, in which case the "negative" is actually valid. We therefore keep Tier 3 edits only where the ground-truth answer under the edited image is known to differ. Geometry3K-style problems with formal annotations let us recompute the answer programmatically. We discard any edit whose answer cannot be checked, and we report what fraction of edits survives this filter.

Training loop. We run GRPO as in VL-Cogito and Infi-MMR. For every prompt whose group is all-correct (zero advantage), plus a sampled fraction of correct rollouts from mixed groups, we compute $\mathcal{L}_{\text{neg}}$. The tier controller keeps a per-prompt running estimate of how often the hinge is active. It promotes a prompt to a harder tier when that rate falls below about 0.3, and demotes it when the rate exceeds about 0.7. Keeping the lower tiers in rotation guards against the forgetting that discrete stage transitions cause.

Testbeds.

  • Primary: Qwen2.5-VL-3B, the Infi-MMR base, for direct comparability.
  • Secondary: an early-fusion unified decoder in which image tokens enter the same transformer and the same generation cache as text.

The secondary testbed tests the claim that the rollout-to-prefill cost ratio, and therefore our advantage at equal compute, grows when visual tokens live inside the policy's own cache.

Accounting. We log GPU-hours per update, split between rollout decoding, reward checking, and negative prefills. All comparisons are made at matched wall-clock time, not matched steps.

Baselines

Plain GRPO. The reference point is plain GRPO on the same data, model, and prompt budget, with no curriculum. For scale, VL-Cogito reports its vanilla GRPO baseline at 58.1% average across its 10 benchmarks, rising to 58.9% with its curriculum and 59.5% with the dynamic length reward. Those are 7B numbers, so they give scale but not a direct target at 3B.

VL-Cogito-style curriculum RL. The strongest existing method on our exact question is curriculum RL with online difficulty weighting, as in VL-Cogito. We reimplement it on the 3B base with identical prompts, rollout count, and tuning budget, and give the extra compute our method spends on prefills back to it as additional rollouts. At 3B the reported reference is Infi-MMR:

  • 67.2% on MathVista testmini
  • 43.68% on MathVerse testmini
  • 27.04% on MathVision

Beating this baseline means, at matched GPU-hours, a gain of at least 2 points on MathVerse's vision-dominant and vision-only versions, with no loss on MathVista testmini. It also means a smaller gap between text-dominant and vision-only accuracy.

Rollout-efficiency baseline. This is SPO-style prioritized prompt sampling from Single-stream Policy Optimization, which avoids degenerate groups altogether. SPO reports a 4.35x throughput gain and +3.4 pp maj@32 over GRPO on text math with Qwen3-8B. There is no reported multimodal result for SPO in the file.

Ablations. Three ablations isolate the key ingredient:

  • a fixed logistic tier schedule in the BACL style instead of the margin controller;
  • Tier 0 only, which reduces to a with-versus-without-image symmetry check;
  • the same contrastive term applied to a fixed offline set of correct solutions instead of fresh on-policy rollouts.

Why not just…?

Why not just run SCPO-style preference optimization after RL? This is the most serious threat. SCPO reduced LLaVA-v1.5-7B's hallucination score (CHAIRs) from 54.7 to 20.3 using offline curriculum pairs and a symmetry loss. A two-stage pipeline of RL followed by SCPO could match us. We run it at matched total compute. If it closes the vision-only gap as well as our method does, then the on-policy recycling claim fails and we will say so. Our bet is that offline pairs go stale as the policy's reasoning style drifts under RL, which SCPO itself acknowledges by resetting its reference model at each curriculum transition.

Why not just skip degenerate groups or sample prompts near 50% success? This fixes wasted compute but adds no pressure toward using the image. A policy that answers correctly from text shortcuts has high success and is simply sampled less often. The SPO-sampling baseline tests this directly. It should match our throughput but not our vision-only gain.

Why not just penalize answers that stay correct when the image is removed? Doing so requires an extra rollout per check, which doubles generation cost, and it only covers our easiest tier. Our Tier-0-only ablation is the cheap version of this idea. Comparing it with full tiers shows whether harder, near-miss images add anything beyond "does the image matter at all."

Why not just use supervised training on the contrastive term with no RL? The term needs correct reasoning chains to exist in the first place, and RL is what produces them from the policy. VL-Cogito reports +5.5 points on MathVista from RL without supervised warm-up. The offline-positives ablation measures how much of our gain survives without fresh rollouts.

Proposed Experiments and Decisive Checks

Training data and evaluation. We train on the mixed-difficulty multimodal math pools used by VL-Cogito and Infi-MMR, with Geometry3K supplying programmatically checkable Tier 3 edits. We evaluate on:

  • MathVerse testmini, broken down by all six versions;
  • MathVista testmini;
  • MathVision;
  • LogicVista;
  • POPE and HallusionBench, to test whether image dependence carries over beyond math.

Decisive check 1: vision dependence. On MathVerse, we measure the accuracy difference between text-dominant and vision-only versions. Our method should shrink this difference relative to all baselines at matched compute. If the difference does not shrink, the method is not doing what it claims, whatever the headline accuracy.

Decisive check 2: cost. We plot accuracy against GPU-hours for three conditions: rollouts only, rollouts plus negatives, and rollouts plus extra rollouts. We repeat this for both testbeds. We predict that the advantage of negatives over extra rollouts is larger for the unified decoder. If it is not, the architectural part of the claim is dropped.

Decisive check 3: margin transfer. We measure the margin $m_3$ on held-out edited diagrams. It should track vision-only accuracy gains. If margins grow while accuracy does not, the model has learned to detect image edits rather than to reason from the diagram.