When a vision-language model is trained with reinforcement learning on math and diagram problems, it samples a group of attempts per problem and is rewarded when the final answer checks out. Problems where every attempt fails contribute nothing. Every attempt gets the same zero reward, so there is no contrast to learn from. These are exactly the problems the model most needs to learn. This proposal recovers that lost signal without a teacher solution. For each all-fail problem, we find the earliest step where the model's own attempts went wrong. We check whether that step misread the image or reasoned badly from a correct reading. We then turn it into a short warning, for example "the diagram does not show AB parallel to CD; do not assume it," and resample with the warning in view. Classical search calls such a warning a heuristic censor: a learned rule that blocks a move known to lead to a dead end. If the warned resamples now include some successes, the problem has moved from "unlearnable" to "learnable." The policy is then trained on those successes with the warning removed, so it learns to avoid the mistake by itself.
This reframes a live disagreement about curricula, the practice of ordering training problems by difficulty. Some multimodal work insists on easy-before-hard ordering. Other work finds that easy problems actively hurt. We argue both sides are measuring the wrong thing. What matters is whether a problem produces mixed outcomes within a sampled group. Censors are a way to move hard problems into that zone on demand, and the same censors can prune sibling branches at test time.
Core Claim
The claim has two parts, and each can be tested. The first is that typed, self-derived failure censors convert a substantial fraction of all-fail training problems into mixed-outcome problems. The second is that training on the converted problems improves held-out multimodal reasoning beyond what difficulty-ordered curricula and difficulty-filtered data achieve at matched compute. "Typed" means each censor is labeled as perceptual (the step contradicts what the model itself sees when asked directly about the image) or deductive (the visual facts were right but the inference was not). This matters because the two failure kinds call for different fixes. They also correspond to the split that MATH-Vision found in GPT-4V's errors, 31.9% vision recognition errors versus 42.2% reasoning errors.
Where This Sits
Three lines of prior work motivate the proposal.
The first is classical explanation-based learning. Adaptive Search by Explanation-Based Learning of Heuristic Censors showed that deliberately over-general failure explanations prune search sharply: 60 versus 99 expanded states in Blocks World, and 23 of 27 otherwise unsolvable larger problems solved after transfer. The same paper also constrains our design. It documents two hazards. Over-general censors can block every valid path, which forced its system to add a relaxation mechanism. Large banks of censors can cost more to match than they save, which it calls the utility problem. We respond to both. Our censors are local to a single problem and never pooled into a cross-problem bank. Each censor is also automatically dropped if the warned resamples fail as badly as the unwarned ones.
The second line is multimodal search that ignores negative structure. VisuoThink runs rollouts and votes among candidates, reports that accuracy rises and then falls as tree width grows, and names pruning as future work. PRM-BAS narrows its beam on a fixed schedule guided by a learned step scorer. Failure is Feedback does use failure context to backtrack, revise, and escalate, but over document-retrieval graphs rather than deductive steps, and at a 5–6× latency cost. None of these turns a failed branch into an explanation that changes later sampling, and none trains the policy.
The third line is multimodal curriculum reinforcement learning. Each of these methods treats failed attempts only as scalar zeros.
- ThinkLite-VL uses search iteration counts to keep only hard problems. It finds that easy problems degrade training, then discards the failed search trees it generated.
- VL-Cogito reweights problems online by rollout accuracy across easy, medium, and hard stages.
- RuCL schedules teacher-written rubric rewards that are tiered once at the start of training.
- OpenVLThinker moves from medium to hard data across iterated supervised and RL rounds.
- Learning When to Look rewards the model for re-checking the image after expressions of uncertainty.
We reuse the hard-problem focus of ThinkLite-VL and the online re-estimation of VL-Cogito. What none of them does, and we do, is mine the failed trajectories for content and re-estimate difficulty by failure type.
Research Program
Training starts from Qwen2.5-VL-7B, the base used by VL-Cogito, RuCL, OpenVLThinker, and ThinkLite-VL. It runs group-relative policy optimization (GRPO, which scores each sampled attempt against the others in its group) on the ViRL-39K pool used by RuCL.
Whenever a problem's group is all-fail, three things happen.
The first is localization. Following the rollout-based soft labels of PRM-BAS, we estimate $v_t$, the probability that $m$ continuations sampled from step $t$ of a failed attempt reach the correct answer. We then take the first wrong step as
$$t^\* = \arg\max_t \,(v_{t-1} - v_t),$$
the step with the largest drop in estimated success. To control cost, $t^\*$ is found by binary search over step boundaries rather than by scoring every step.
The second is typing. If step $t^\*$ asserts a visual fact (a label, length, count, or relation), that fact is posed back to the same model as a direct question about the image, sampled several times. Disagreement with the step's claim marks the failure perceptual; otherwise it is marked deductive. The policy then writes a one-sentence censor from the failed step. Perceptual censors name the misread fact and instruct a re-look. Deductive censors name the invalid move. Censors from the problem's distinct failed attempts are merged and capped at a few sentences.
The third is censored resampling. A fresh group is drawn with the censors added to the prompt. If it contains successes, those trajectories enter the GRPO update with the censors stripped from the context. Their likelihood ratio is computed under the uncensored prompt and clipped, so the policy internalizes the avoidance instead of learning to depend on hints. This off-policy step is the most fragile part of the design, so we monitor the clipping rate and the shift in the policy's output distribution, measured as KL divergence. If the warned group does no better than the unwarned one, the censor is discarded; this is our analogue of FS2's relaxation.
The curriculum then follows directly. Every few hundred steps, each problem is re-assigned among three groups. Solved problems are dropped, consistent with ThinkLite-VL. Mixed problems are trained normally. All-fail problems are routed to censored resampling, with perceptual and deductive shares logged per problem. The prediction is that a model's progress is governed by how many problems sit in the mixed zone, and that censors raise that number more cheaply than adding samples.
The same machinery gives a test-time variant. Within a single problem's search tree, a branch that fails a self-consistency or verification check produces a censor, which is applied only to its unexpanded siblings. This keeps matching cost proportional to one tree rather than to a growing bank.
Baselines
The naive reference point is base Qwen2.5-VL-7B with standard chain-of-thought sampling, along with vanilla GRPO on ViRL-39K. To match compute, vanilla GRPO is given the extra rollout tokens our method spends on localization and resampling, as more samples per all-fail problem. This tests the plainest explanation for any gain, which is that more samples on hard problems is enough.
The strongest data-selection competitor is ThinkLite-VL, which reports 75.1% on MathVista at 7B (ThinkLite-VL). Because that result comes from its own filtered pool, we report that number and also rerun its search-based filtering on ViRL-39K under our token budget. The strongest curriculum competitor is VL-Cogito, which reports 74.8% on MathVista, 68.7% on Geometry3K, and 48.9% on LogicVista (VL-Cogito). We rerun its online difficulty weighting on the same pool and budget. RuCL reports 60.06% averaged over seven benchmarks after training on ViRL-39K (RuCL), so it serves as the same-data reference, though it relies on a teacher model we do not use. For WeMath, Learning When to Look reports 54.4 (Learning When to Look). For MATH-Vision at 7B, the file contains no reported number for any of these RL methods; the only 7B figure is 28.95% for Qwen2-VL-7B with RCTS (RCTS). All reruns get equal hyperparameter-search budgets. Beating a baseline means a higher mean over three seeds on the held-out MATH-Vision and LogicVista, by more than the seed spread, while not losing on MathVista.
Three ablations isolate the key ingredient. Untyped censors, where every censor is written as a deductive warning, test whether the perceptual-versus-deductive split matters. Keeping censors in context but never stripping them tests whether the model truly internalizes the avoidance. Reference-solution hints for all-fail problems, at matched token count, test whether self-derived negative knowledge competes with privileged positive knowledge. For the test-time variant, we compare against VisuoThink-style self-voting and PRM-BAS beam annealing at matched generated tokens on Geomverse-109 and Geometry3K. For VisuoThink, the file reports only a relative figure, 17.1% over chain-of-thought on Geomverse-109, so our comparison rests on our own reruns.
Why not just…?
Why not just sample more attempts on hard problems? Plain resampling is the null hypothesis for the conversion claim. Censors earn their place only if they move more all-fail problems into the mixed zone per token than extra samples do. The first decisive check below measures exactly this, and if the two rates match, the proposal reduces to a sampling-budget result.
Why not just give reference-solution hints on all-fail problems? This is a real threat, because ViRL-39K has verified answers and many training sets carry worked solutions. Hints may convert problems faster. Our bet is that hint-conditioned successes are further from what the policy would produce without the hint, so they distill worse once the hint is removed, whereas censors only rule out the model's own mistakes. The matched hint ablation settles this. If hints win on held-out accuracy, we will report that the value lies in conditioning, not in failure content.
Why not just use a step-level process reward that penalizes the first wrong step? PRM-BAS shows step-level value estimates help search, and a penalty at $t^\*$ would reuse our localization at lower cost. A penalty, however, still yields no positive example on all-fail problems, which is where we claim the bottleneck lies. We run this variant with identical localization so that any difference is due to censored resampling itself.
Why not just filter for hard problems, as ThinkLite-VL does? Filtering decides which problems to train on. It does nothing to make an unsolved problem produce useful learning signal, and ThinkLite-VL discards precisely the failed trees we mine. The comparison on the same data shows whether that discarded material is worth the localization cost.
Proposed Experiments and Decisive Checks
The first check is conversion. At a fixed token budget per problem, what fraction of all-fail ViRL-39K problems becomes mixed under censors, under plain resampling, and under reference hints, tracked over training?
The second check is held-out accuracy. We evaluate on MATH-Vision, LogicVista, WeMath, MathVista, and Geometry3K against the baselines above. We pay particular attention to MATH-Vision disciplines such as Transformation Geometry, where GPT-4o reaches only about 18% (MATH-Vision).
The third check validates typing. We hand-label about 300 localized failures as perceptual or deductive and measure agreement with the automatic labels. Low agreement would invalidate the typed curriculum regardless of benchmark gains. As a further sanity check, perceptual censors should help most on problems where Plot Twist-style shortcuts are possible, that is, where models can answer from the question text without reading the image.
The fourth check concerns test-time pruning. We plot accuracy against tree width with and without sibling censors, to see whether pruning flattens VisuoThink's rise-then-fall curve at matched tokens. We also log how often a censor blocks a branch that later proves correct, which is the over-pruning rate.
Taken together, the proposal is supported if censors convert problems more cheaply than resampling, typed beats untyped, stripped beats kept, and censors at least match reference hints on held-out sets. It is refuted if plain resampling or hints match censors.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Adaptive Search by Explanation-Based Learning of Heuristic Censors
- VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Search
- PRM-BAS: Enhancing Multimodal Reasoning through PRM-guided Beam Annealing Search
- Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphs
- SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement
- VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
- RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoning
- OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
- Learning When to Look: A Disentangled Curriculum for Strategic Perception in Multimodal Reasoning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details