Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning

Baolong Bi, Shenghua Liu, Yiwei Wang, Siqian Tong, Lingrui Mei, Yuyao Ge, Yilong Xu, Jiafeng Guo, Xueqi Cheng

Abstract

Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose RGR-GRPO (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.

Introduction

Reinforcement Learning (RL) has recently supercharged the reasoning capabilities of Large Language Models (LLMs)—most notably seen in models like DeepSeek-R1. However, current approaches face two major hurdles:

  1. Sparse and Domain-Limited Rewards: Most RL for LLMs relies on "verifiable rewards" (e.g., checking if a math answer or code snippet is exactly correct). This works for math but fails in open-ended reasoning where "correctness" is nuanced and intermediate steps matter.
  2. The Exploration Bottleneck: Purely "online" RL—where the model only learns from its own current attempts—often suffers from entropy collapse. The model quickly finds one way to solve a problem, stops trying anything else, and gets stuck at a performance ceiling.

The authors aim to move beyond simple "right/wrong" binary feedback by using structured rubrics to guide the model through complex, multi-domain problems.

Methodology: RGR-GRPO

The authors introduce RGR-GRPO (Reward and Guidance through Rubrics), which builds on the Group Relative Policy Optimization (GRPO) algorithm. It introduces two novel components:

  • Rubric-Based Fine-Grained Rewards: Instead of a single 0 or 1, the model is judged against two types of criteria: Factual Criteria (verifying intermediate sub-answers) and Process Criteria (evaluating logical steps). These rubrics are generated by an "expert" model (OpenAI o3) and used by a "judge" model (Qwen-32B) to provide dense, scalar rewards.
  • Rubric-Guided Offline Exploration: If the model fails to find a perfect solution during its online attempts, the system performs an Exploration Assessment (EA). It identifies specific failed rubric items, prompts the model to self-refine based on those failures, and uses that "refined" trajectory as an off-policy training signal.
  • Adaptive Variance Control: To prevent training instability, the EA mechanism only triggers high-variance off-policy updates when the model's current exploration is insufficient to reach a correct answer.
The RGR-GRPO framework utilizes rubrics for both rewards and guidance to enhance exploration.

Experimental Setup

The researchers used a diverse array of datasets to prove the model's versatility:

  • Training: A filtered subset of WebInstruct-Verify (~10k samples) covering physics, chemistry, social sciences, and finance.
  • Evaluation: 14 benchmarks across categories including Math (MATH, SMath), Physics (PIQA, SPhys), Chemistry (Chem, SChem), and General Reasoning (MMLU-Pro, GPQA). Out-of-Distribution (OOD) tasks included MedMCQA and CS-Bench.

Results and Discussion

The RGR-GRPO framework consistently outperformed both base models and standard RL baselines:

  • Significant Gains: On the Qwen2.5-7B model, it achieved average improvements of +7.0% in math, +5.4% in physics, and +8.4% in chemistry compared to verifiable online RL.
  • Superior Exploration: The model showed much higher pass@k scores, reflecting sustained exploration. At k=64 on Sci-Math, RGR-GRPO reached 93.2%, significantly higher than the base model's 25.2%.
  • Generalization: Despite being trained mostly on math and science, the model showed strong "zero-shot" improvements on medical (MedMCQA) and CS (CS-Bench) tasks, suggesting it learned how to reason, not just how to solve specific formulas.
Performance comparison across domains and Pass@k metrics on SciBench, showing RGR-GRPO's superior reasoning and exploration.

Conclusion

RGR-GRPO effectively breaks the exploration bottleneck in LLM reasoning by using structured rubrics as a bridge between rule-based checkers and flexible human-like reasoning. By forcing "self-refinement" through failed rubrics, the framework pushes the model's frontier into new areas of the solution space while maintaining stability through selective off-policy updates. Future work will explore automated rubric generation and application to more open-ended domains like creative writing or legal reasoning.

Summary updated

Abstract

Recent advances in reinforcement learning (RL) have significantly improved the complex reasoning capabilities of large language models (LLMs). Despite these successes, existing methods mainly focus on single-domain RL (e.g., mathematics) with verifiable rewards (RLVR), and their reliance on purely online RL frameworks restricts the exploration space, thereby limiting reasoning performance. In this paper, we address these limitations by leveraging rubrics to provide both fine-grained reward signals and offline guidance. We propose $\textbf{RGR-GRPO}$ (Reward and Guidance through Rubrics), a rubric-driven RL framework for multi-domain reasoning. RGR-GRPO enables LLMs to receive dense and informative rewards while exploring a larger solution space during GRPO training. Extensive experiments across 14 benchmarks spanning multiple domains demonstrate that RGR-GRPO consistently outperforms RL methods that rely solely on alternative reward schemes or offline guidance. Compared with verifiable online RL baseline, RGR-GRPO achieves average improvements of +7.0%, +5.4%, +8.4%, and +6.6% on mathematics, physics, chemistry, and general reasoning tasks, respectively. Notably, RGR-GRPO maintains stable entropy fluctuations during off-policy training and achieves superior pass@k performance, reflecting sustained exploration and effective breakthrough beyond existing performance bottlenecks.
BibTeX
@inproceedings{bi2026reward,
  author = {Baolong Bi and Shenghua Liu and Yiwei Wang and Siqian Tong and Lingrui Mei and Yuyao Ge and Yilong Xu and Jiafeng Guo and Xueqi Cheng},
  title = {{Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=AfqsNFzJcs},
}
Figure