Reinforcement learning on problems with checkable answers only teaches a model something when the model sometimes succeeds and sometimes fails on a problem. If every attempt succeeds, or every attempt fails, the training signal is exactly zero. For vision-language models solving diagram-based math, the usual fix is to make variants of each problem: easier versions when the model is stuck, harder ones when it has mastered the original. Today those variants are expensive or crude. Easier versions come from querying a strong external model for hints. Harder versions come from blurring the image. In both cases a new answer may then need to be checked.
We propose a cheaper and more meaningful dial. A diagram problem states some facts in the text ("AB = 5") and others only in the picture (a length label, a right-angle mark, a bar height). To make the problem easier, copy some picture-only facts into the text. To make it harder, delete facts from the text and draw them into the diagram. These moves only relocate information, so the correct answer never changes and no variant needs to be verified again. The dial also points directly at the cross-modal gap: models reason far better over facts they are told in words than over facts they must read off pixels. Turning the dial therefore controls exactly the skill the model lacks. If the dial is always set where the model succeeds about half the time, training should follow an automatic schedule. It starts with text-heavy variants and drifts toward image-only problems as the model learns to read diagrams.
Core Claim
Answer-preserving relocation of facts between modalities gives a continuous, verification-free difficulty control for multimodal policy-gradient training. Keeping each problem at intermediate success with this control should match or beat teacher-hint variant generation on visual math benchmarks, at a small fraction of the synthesis cost. It should also shrink the gap between how the model performs when information is given in text and when it is given only in the image. The new element is not difficulty-targeted sampling, which already exists. It is a difficulty axis that is semantic, cheap, and inherently cross-modal, together with the measurable prediction that the frontier along this axis drifts toward vision-only problems during training.
Where This Sits
Three lines of work motivate the proposal. First, the gradient argument. In Group Relative Policy Optimization (GRPO), the method that scores each sampled answer against the average of its group, groups with identical outcomes yield zero advantage. Advantage Collapse in Group Relative Policy Optimization documents this collapse. CurES derives that update strength peaks near 50% accuracy and tracks per-prompt difficulty with Beta posteriors. Knapsack RL argues the peak can sit nearer one third when exploration budgets are tight. We reuse the Bayesian tracking and test both targets. Unlike CurES, we do not select from a fixed pool. We move each problem along a dial.
Second, difficulty-adaptive synthesis. SwS synthesizes new text problems with 70B/32B models and discards anything outside 25–75% student accuracy, accepting large rejection waste. HeaPA cuts overhead in text math with student-side numeric perturbations and a small asynchronous verifier. It reports 51.1% vs 46.5% average accuracy over GRPO on DAPO-Math with about 2.1% wall-clock overhead. Its lineage-aware smoothing of correlated variants informs our per-problem difficulty model. Both papers are text-only, and both still need a verifier because their variants change the answer. Ours do not.
Third, and closest, DIVA-GRPO brings difficulty-adaptive variants to multimodal math. It asks GPT-o3 for step hints when problems are too hard and adds blur or salt-and-pepper noise when they are too easy. It reports 2.55× fewer training steps and 1.76× less wall-clock time than GRPO. Our proposal differs on both sides of its dial. Our "easier" move needs no teacher call per variant. Our "harder" move removes stated information instead of corrupting pixels, so it tests diagram reading rather than robustness to noise.
On the cross-modal side, PaRe supplies the structural precedent. It anneals inputs from a familiar source modality into a target modality by patch replacement, and shows this smooths optimization compared with two-stage transfer. It does so for supervised fine-tuning with a fixed linear schedule. We transplant the idea to policy-gradient training, with two changes: the schedule is set by the student's success rate instead of a clock, and the unit being replaced is a semantic fact instead of a patch. Two papers constrain the design. AMLRIS shows that uninformed image regions inject misleading gradients. PaRe reports that random replacement adds noise. Both argue against blur-style difficulty. SynTIC and DisCL bridge synthetic-to-real gaps with diffusion-generated images. Their authors flag generation cost as a bottleneck, which is why we avoid generative image synthesis entirely and only render text labels onto existing diagrams.
Research Program
Construction happens once per seed problem. A mid-sized open vision-language model extracts a small "fact sheet" of atomic visual facts from each diagram (labeled lengths, angles, values, relations). We keep a fact only if a reference solver given the text-augmented problem still reaches the dataset answer. This is the only verification step, and it is paid once per problem, not once per variant. We also parse the text into atomic givens that can be rendered as labels at known diagram locations. Rendering is done with an ordinary image library; no diffusion is involved.
This defines a dial position $\ell \in [-1, 1]$. Negative values move a fraction $|\ell|$ of text givens into the image. Positive values move a fraction $\ell$ of extracted visual facts into the text. The answer is identical at every $\ell$.
During training, each problem gets a two-parameter success model fitted from its rollouts at whatever dial positions it has visited:
$$p_j(\ell) = \sigma(a_j + b_j \ell),$$
where $p_j(\ell)$ is the student's success probability on problem $j$ at dial position $\ell$, $\sigma$ is the logistic function, and $a_j, b_j$ are fitted online by Bayesian logistic regression. Pooling across positions is what makes this efficient. A single group of rollouts at one setting informs the whole curve, which addresses the tracking-lag and correlated-variant problems that HeaPA handles with lineage aggregation. At each visit we set $\ell_j$ to the value where $p_j(\ell_j) = p^*$, with $p^* \in \{1/3, 1/2\}$. Problems whose fitted curve never reaches $p^*$, even fully textualized, are deferred rather than retried. This guards against the 0%-success trap.
The analysis has two parts. Behaviorally, on held-out problems we measure the modality gap as accuracy at $\ell = +1$ (facts in text) minus accuracy at $\ell = -1$ (facts only in the image). Representationally, we compare the model's hidden states at the answer position under the two versions to see whether the same problem converges to the same internal state. We also log the average chosen $\ell$ over training. The central mechanistic prediction is that this average drifts downward, from text-heavy toward image-only, with no schedule imposed.
Baselines
The naive reference is standard GRPO with uniform sampling over the same seed pool and the same rollout budget. It shows how much of the signal is lost to all-correct and all-wrong groups.
The strongest selection-only competitor is CurES-style Beta-posterior prompt selection near 50%, run on the same pool with the same rollouts per step. It isolates whether synthesis helps at all beyond picking the right originals.
The strongest existing method is DIVA-GRPO: teacher hints for hard problems, visual or text noise for easy ones, plus its advantage rescaling. We run it on the same base 7B vision-language model, the same seed problems, and the same number of policy rollouts. Because GPT-o3 may be unavailable or unaffordable, the teacher is a fixed open model, and the identical model also serves as our fact extractor. We give it an equal number of tuning runs. We will reproduce its headline efficiency figures (2.55× fewer steps, 1.76× less wall-clock than GRPO). For absolute benchmark accuracy, no reported result was found on Lacuna for DIVA-GRPO's MathVista, MathVerse, MathVision, WeMath, or MMK12-test scores. The file states only that it beats Qwen2.5-VL-32B and InternVL2.5-VL-78B. Comparison will therefore rest on our own runs.
We count as beating DIVA-GRPO either of two outcomes: matching its average across the five benchmarks while spending at least five times less synthesis compute, measured as GPU-hours plus teacher tokens per training group with nonzero advantage; or exceeding it by a margin larger than seed-to-seed variation across three seeds at equal compute.
Three ablations isolate the key ingredient. The first uses one-directional dials, easier-only or harder-only. The second sets $\ell$ at random instead of targeting $p^*$, which tests whether calibration matters or just exposure to variants. The third replaces the success-driven $\ell$ with a fixed linear schedule from $+1$ to $-1$, which is a direct transplant of PaRe's annealing.
Why not just…?
Why not just keep DIVA-GRPO's teacher hints? Hints may genuinely be more powerful than relocating facts. A hint can supply a reasoning step that no visual fact provides, so on problems that fail for logical rather than perceptual reasons, our dial cannot help. This is a real threat. We will split held-out failures by whether the fully textualized version is solved. On "reasoning-bound" problems we expect hints to win. On "perception-bound" problems we expect the dial to win. A hybrid that calls the teacher only for problems whose fitted curve never reaches $p^*$ is the natural fallback, and its teacher-call count is the quantity to report.
Why not just select originals near 50% accuracy, as CurES does, with no variants? Selection cannot rescue a problem that sits at 0% or 100%; it can only skip it. It is cheap, but on a fixed pool the supply of intermediate problems shrinks as the model improves. The CurES-style baseline at equal rollouts answers this directly. If it matches us, the dial adds nothing, and we will say so.
Why not just always train with captions or textualized facts? That makes problems easier but likely teaches the model to depend on text. The prediction is that it improves text-dominant accuracy and leaves vision-only accuracy unchanged, which would show up as an unchanged or wider modality gap. The fixed-$\ell = +1$ run tests exactly this.
Why not just add blur and noise to make easy problems harder? AMLRIS and PaRe both report that uninformed pixel manipulation produces misleading gradients. More to the point, a blurred diagram can destroy information needed for the answer, so the label silently becomes wrong. The harder-only ablation set against DIVA's noise branch compares the two directly.
Proposed Experiments
We train a 7B vision-language model on a pool of diagram-based math problems. We use the multimodal training data DIVA-GRPO uses if it is obtainable, and otherwise a comparable diagram math corpus disjoint from all test sets. Evaluation covers MathVista, MathVerse, MathVision, WeMath, and MMK12-test. Text-only OlympiadBench serves as a check that text reasoning is not harmed. We report accuracy, training steps and wall-clock time to reach the GRPO peak, the fraction of groups with zero advantage, and synthesis compute per useful group.
Four outcomes are decisive. First, the frontier position $\ell$ chosen by the controller must drift toward image-only during training. If it stays text-heavy, the model is not learning to read diagrams and the core story fails. Second, the held-out modality gap must shrink more under our method than under DIVA-GRPO or always-caption training, and hidden-state agreement between the text and image versions should rise with it. Third, the comparison of $p^* = 1/3$ and $p^* = 1/2$ asks whether perception-bound difficulty follows the same optimum as symbolic reasoning, a question the file identifies as open. Fourth, we audit 200 sampled variants by hand to confirm the answer is truly preserved. If more than a few percent of relocated facts are wrong, the "verification-free" claim does not hold, and the one-time solver check must be tightened before any accuracy claim stands.
References
- Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
- CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
- Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
- SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
- HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning
- DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage
- Enhancing Cross-Modal Fine-Tuning with Gradually Intermediate Modality Generation
- AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation
- Improving Cross-Modal Alignment with Synthetic Pairs for Text-Only Image Captioning
- Diffusion Curriculum: Synthetic-to-Real Generative Curriculum Learning via Image-Guided Diffusion