Abstract
Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models will be released and open-sourced.
Introduction
Reinforcement Learning (RL) is the primary engine behind recent leaps in LLM reasoning capabilities. However, RL requires a high-quality reward signal to guide the model. While rewards are easily verifiable in domains like mathematics or coding, general conversation lacks such objective ground truths. Traditional Scalar Reward Models act as black boxes, outputting a single number that lacks transparency and, crucially, fails to scale effectively with additional inference-time compute.
This paper introduces DeepSeek-GRM, a generalist reward model that achieves state-of-the-art performance by shifting from scalar outputs to Pointwise Generative Reward Modeling (GRM). By allowing the model to "think" before issuing a reward—generating principles and critiques—we enable a new paradigm of inference-time scalability for reward signals.
Methods: Self-Principled Critique Tuning (SPCT)
To train DeepSeek-GRM, the authors developed Self-Principled Critique Tuning (SPCT). This method ensures the model doesn't just generate text, but generates useful reasoning that leads to accurate rewards. The process involves two key stages:
- Rejective Fine-Tuning (RFT): A "cold start" phase where the model is trained on high-quality trajectories of principles and critiques, filtered to ensure predicted rewards align with ground-truth labels.
- Rule-Based Online RL: Using the GRPO (Group Relative Policy Optimization) algorithm, the model is refined. It receives a positive reward only if its generated reasoning leads to a score that correctly identifies the "best" response among candidates.
Inference-Time Scaling
A core contribution of this work is the demonstration that reward models can benefit from "test-time compute" just like reasoning models. DeepSeek-GRM uses parallel sampling to generate $k$ independent sets of principles and critiques. These are aggregated using two methods:
- Voting: Averaging the scores from $k$ samples.
- Meta RM Guided Voting: A separate, lightweight "Meta RM" is trained to judge the correctness of the generated critiques. This acts as a filter, selecting the highest-quality reasoning traces to determine the final reward.
Key Results and Scaling Laws
DeepSeek-GRM-27B was evaluated on a comprehensive suite of benchmarks, including Reward Bench (RB) and PPE. The results demonstrate a clear log-linear relationship between the number of samples ($k$) and reward accuracy.
As shown in the scaling analysis, DeepSeek-GRM-27B (MetaRM@32) achieved an overall score of 72.8, surpassing proprietary giants like GPT-4o (71.3) and massive open models like Nemotron-4-340B (70.5). This suggests that scaling inference compute is more efficient than scaling training compute; a 27B model with 32 samples can outperform a 671B model using greedy decoding.
Conclusion
DeepSeek-GRM proves that generative reward modeling, combined with proper learning methods like SPCT, enables effective inference-time scaling for generalist queries. While generative RMs introduce higher latency than scalar models, the gains in accuracy, transparency, and bias reduction represent a significant step forward for RL in non-verifiable domains. Future work will explore extending this approach to Process Reward Modeling (PRM) to reward individual steps of reasoning chains.