TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) helps models get the right answer, but it often encourages them to cheat by using flawed logic or "hallucinated" evidence. This paper introduces Groupwise Ranking Reward, a method that compares multiple correct reasoning paths for the same problem and rewards the most logical ones, boosting reliability-conditioned accuracy from 47.4% to 54.7%.
What Problem Does This Paper Address?
In the race to improve Multimodal Large Language Models (MLLMs), researchers often use RLVR, which rewards a model simply for getting the final answer right (e.g., a math problem or a visual puzzle). However, the authors identify a critical failure mode: reasoning-answer inconsistency.
As models are trained longer with outcome-only rewards, they often start "getting the right answer for the wrong reasons." For example, a model might correctly solve for $X=10$ in its final output, even though its internal reasoning steps derived $X=-11$.
This gap makes MLLMs dangerous for high-stakes applications like medical diagnosis or scientific analysis, where the process is just as important as the result.
Methods
The authors compare three ways to supervise the reasoning process (trajectory supervision) within a standard GRPO (Group Relative Policy Optimization) pipeline:
- Process Reward Models (PRMs): These assign scalar scores to every intermediate step. While efficient, they often struggle to generalize as the model's policy shifts during training.
- Pointwise Generative Rewards (GRs): An LLM-as-a-judge evaluates each reasoning path independently against a rubric. This is computationally expensive and suffers from inconsistent scoring across different calls.
- Groupwise Ranking Reward (Proposed): Instead of scoring trajectories in isolation, this method takes all trajectories that passed the answer verifier for a single prompt and ranks them side-by-side in one pass.
The ranking is then converted into a centered reward score. This "redistribution" mechanism effectively penalizes weaker reasoning even if the answer is correct, forcing the model to prefer better-grounded paths.
Datasets
- Training: The models were trained on ViRL, a multimodal image-grounded RL corpus.
- Evaluation: The authors used a diverse suite of five benchmarks:
- Math-focused: MathVision, MathVista, and WeMath.
- General reasoning: MMMU and MMMU-Pro.
- Ablation: They also compared performance against the text-only GSM8K dataset to see how multimodality affects inconsistency.
Key Insights and Ideas
- RLVR is a Double-Edged Sword: Outcome-only rewards actually increase inconsistency over time. While accuracy goes up, the "Correct-But-Inconsistent Rate" (CBIR) climbs steadily, eventually causing reliability to tank.
- Comparison is More Stable Than Scoring: Groupwise ranking reduces reward variance by 37.6% compared to pointwise scoring. By judging trajectories together, the LLM-judge applies a more consistent standard.
- The Mapping Function is Secondary: The authors tested various ways to turn a rank into a score (linear, exponential, etc.). They found that the specific math doesn't matter as much as the act of groupwise comparison itself.
- Scaling Doesn't Solve the Root Cause: While larger models (7B vs 3B) have lower baseline inconsistency, both sizes show the same upward drift in flawed reasoning when trained with outcome-only rewards.
Key Results
- Superior Reliability: Groupwise Ranking Reward achieved the best overall performance, reaching 54.7% Reliability-Conditioned Accuracy (RC-Acc), a significant jump over the 47.4% achieved by standard RLVR.
- Broad Gains: The method wasn't just better on average; it performed best on four out of the five tested benchmarks, showing particularly strong gains in complex visual math (WeMath).
- Efficiency: Because it ranks multiple trajectories in a single LLM call, it requires less judge computation than pointwise methods while providing a cleaner optimization signal.
Limitations and Future Work
- Verifiable Domains Only: This study focuses on tasks with clear, verifiable answers (like math). Applying this to "softer" tasks like creative writing or open-ended dialogue remains an open challenge.
- Judge Blind Spots: The text-only judge used for evaluation sometimes flags correct visual reasoning as inconsistent because it cannot "see" the image-to-letter mappings in multiple-choice questions.
- Future Direction: The authors suggest that future work should focus on better handling visually grounded option mappings and late-stage answer revisions where a model corrects itself mid-thought.
Summary updated