Abstract
Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.
Introduction
Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning Large Language Models (LLMs) with human values. Central to this process is the Reward Model (RM), which provides the scalar signal used to optimize the policy. However, traditional RMs face a significant "bottleneck": they are prone to "reward hacking"—where the actor model exploits model inaccuracies to achieve high scores—and they often generalize poorly to out-of-distribution (OOD) data.
While strong LLM judges (e.g., DeepSeek-R1) offer superior accuracy and robustness through Chain-of-Thought reasoning, their high inference cost and latency make them impractical for every iteration of online RLHF. This paper proposes an uncertainty-based routing framework that leverages a cheap RM for confident predictions and reserves the expensive LLM judge for cases where the RM is uncertain.
Methodology: Uncertainty-Based Routing
The core of the framework is a specialized Pairwise Preference Model (PM). Unlike standard pointwise RMs that output a single score, the pairwise approach models the probability that one response is preferred over another. This formulation avoids the mathematical "indefiniteness" of pointwise models, where adding a constant to rewards doesn't change preferences, making uncertainty quantification more stable.
To quantify uncertainty, the authors equip the PM with a Spectral-normalized Neural Gaussian Process (SNGP) head. This allows the model to estimate epistemic uncertainty—the uncertainty arising from a lack of training data for a specific input.
The routing logic is straightforward: during training, if the uncertainty metric $u$ exceeds a predefined threshold, the pair is routed to the LLM judge. Otherwise, the fast PM's prediction is used. This hybrid signal is then integrated into policy gradient methods like RLOO or GRPO for downstream alignment.
Experimental Setup
The Reward Model was trained on the HelpSteer2-Preference dataset. For RLHF alignment, a Llama-3.1-8B policy was fine-tuned using prompts from Ultrafeedback. The system was evaluated on:
- RewardBench & RM-Bench: To test the RM's robustness and ability to handle difficult reasoning or safety tasks.
- Downstream Benchmarks: Arena-Hard-v0.1, AlpacaEval 2.0, and MT-Bench to measure the final quality of the aligned models.
Key Results
The uncertainty-based router demonstrates significant efficiency gains over random sampling. On the RM-Bench, routing improved average accuracy from 68.0% to 81.9% by calling the judge for 44% of samples, consistently outperforming random routing at equivalent cost levels.
In downstream RLHF tasks, the uncertainty-routed model achieved a 29.8% win rate on Arena-Hard, surpassing both the standard RM (28.1%) and the base model (24.5%). Notably, these gains were achieved while keeping LLM judge calls between 6% and 18% during online training, proving the framework's practical utility.
Conclusion
This work addresses the reward model bottleneck by intelligently routing difficult or OOD examples to a strong LLM judge. By using a pairwise preference model with SNGP, the system achieves a principled measure of uncertainty that correlates strongly with model error. Future work may involve dynamic thresholding for fixed computational budgets or replacing the large LLM judge with smaller, specialized generative RMs to further reduce latency.
Summary updated