Abstract
Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose RGR-GRPO (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.
Introduction
Reinforcement Learning (RL) has recently supercharged the reasoning capabilities of Large Language Models (LLMs)—most notably seen in models like DeepSeek-R1. However, current approaches face two major hurdles:
- Sparse and Domain-Limited Rewards: Most RL for LLMs relies on "verifiable rewards" (e.g., checking if a math answer or code snippet is exactly correct). This works for math but fails in open-ended reasoning where "correctness" is nuanced and intermediate steps matter.
- The Exploration Bottleneck: Purely "online" RL—where the model only learns from its own current attempts—often suffers from entropy collapse. The model quickly finds one way to solve a problem, stops trying anything else, and gets stuck at a performance ceiling.
The authors aim to move beyond simple "right/wrong" binary feedback by using structured rubrics to guide the model through complex, multi-domain problems.
Methodology: RGR-GRPO
The authors introduce RGR-GRPO (Reward and Guidance through Rubrics), which builds on the Group Relative Policy Optimization (GRPO) algorithm. It introduces two novel components:
- Rubric-Based Fine-Grained Rewards: Instead of a single 0 or 1, the model is judged against two types of criteria: Factual Criteria (verifying intermediate sub-answers) and Process Criteria (evaluating logical steps). These rubrics are generated by an "expert" model (OpenAI o3) and used by a "judge" model (Qwen-32B) to provide dense, scalar rewards.
- Rubric-Guided Offline Exploration: If the model fails to find a perfect solution during its online attempts, the system performs an Exploration Assessment (EA). It identifies specific failed rubric items, prompts the model to self-refine based on those failures, and uses that "refined" trajectory as an off-policy training signal.
- Adaptive Variance Control: To prevent training instability, the EA mechanism only triggers high-variance off-policy updates when the model's current exploration is insufficient to reach a correct answer.
Experimental Setup
The researchers used a diverse array of datasets to prove the model's versatility:
- Training: A filtered subset of WebInstruct-Verify (~10k samples) covering physics, chemistry, social sciences, and finance.
- Evaluation: 14 benchmarks across categories including Math (MATH, SMath), Physics (PIQA, SPhys), Chemistry (Chem, SChem), and General Reasoning (MMLU-Pro, GPQA). Out-of-Distribution (OOD) tasks included MedMCQA and CS-Bench.
Results and Discussion
The RGR-GRPO framework consistently outperformed both base models and standard RL baselines:
- Significant Gains: On the Qwen2.5-7B model, it achieved average improvements of +7.0% in math, +5.4% in physics, and +8.4% in chemistry compared to verifiable online RL.
- Superior Exploration: The model showed much higher pass@k scores, reflecting sustained exploration. At k=64 on Sci-Math, RGR-GRPO reached 93.2%, significantly higher than the base model's 25.2%.
- Generalization: Despite being trained mostly on math and science, the model showed strong "zero-shot" improvements on medical (MedMCQA) and CS (CS-Bench) tasks, suggesting it learned how to reason, not just how to solve specific formulas.
Conclusion
RGR-GRPO effectively breaks the exploration bottleneck in LLM reasoning by using structured rubrics as a bridge between rule-based checkers and flexible human-like reasoning. By forcing "self-refinement" through failed rubrics, the framework pushes the model's frontier into new areas of the solution space while maintaining stability through selective off-policy updates. Future work will explore automated rubric generation and application to more open-ended domains like creative writing or legal reasoning.
Summary updated