Beyond Accuracy: Measuring Reward Variance as a Predictive Benchmark for RLHF

Jinnan Li, Yuxing Cheng, Yue Wang, Yuan Wu, Yi ChangOpenReview 2026 · Submission to ICLR 2026 · Rejected · OpenReview

TL;DR

This paper argues that pairwise accuracy is an insufficient metric for evaluating Reward Models (RMs) in RLHF. The authors introduce the Reward Variance Benchmark (RVB), which measures how well an RM concentrates, separates, and stabilizes its scores to provide actionable gradients for policy optimization. Their findings show that reward variance is a better predictor of downstream RLHF convergence than accuracy alone.

What Problem Does This Paper Address?

In Reinforcement Learning from Human Feedback (RLHF), the Reward Model acts as the "teacher," providing the signal that guides a Large Language Model (the "student") toward human-preferred behavior. Traditionally, we evaluate these teachers based on pairwise accuracy: if they can correctly identify which of two responses is better.

However, the authors point out an "accuracy paradox": sometimes a more accurate RM leads to worse policy performance. The missing link is variance. If an RM is accurate but assigns nearly identical scores to all responses, the optimization landscape becomes "flat." The policy gradient shrinks, and the model learns incredibly slowly. To be an effective teacher, an RM doesn't just need to be right; it needs to be decisive and consistent.

Methods

The authors propose the Reward Variance Benchmark (RVB), which moves beyond simple rankings to analyze the distribution of reward scores. Key methodological contributions include:

  1. Robust Standardization: Since different RMs output scores on wildly different scales, the authors use a robust standardization process based on the Median Absolute Deviation (MAD) to put all models on a level playing field.
  2. The Metric Suite: They introduce three core metrics to profile an RM's "teaching signal":
  3. SEI (Softmax-Entropy Index): Measures concentration. Does the model clearly favor a few candidates (peaky) or is it unsure (flat)?
  4. nGMD (normalized Gini mean difference): Measures global separation. How well does the model spread out the scores across a set of candidates?
  5. DCI (Decision Consistency Index): Measures stability. Does the model maintain a consistent scoring style across different prompts, or is it erratic?
  6. Composite Scoring: A unified index that balances these three dimensions to rank the overall "teaching effectiveness" of an RM.

Datasets

The benchmark is built on Eval-Core, a distilled subset of the Reward Model Benchmark (RMB).

  • Source: 1,906 prompts covering 12 task categories.
  • Candidates: For each prompt, the authors generated 15–17 responses using GPT-4o at various temperatures to create a controlled "quality gradient."
  • Eval-Core: A high-quality subset of 354 prompts with 9 responses each, specifically sampled to ensure a mix of high and low consensus among existing RMs.
  • Eval-Semantic: An auxiliary dataset used to test "semantic robustness"—whether an RM gives the same score to two different sentences that mean the same thing.

Key Insights and Ideas

  • Accuracy $\neq$ Training Signal: Accuracy and reward variance are only moderately correlated ($r \approx 0.51$). Some top-tier models on accuracy leaderboards (like ArmoRM) rank surprisingly low on variance metrics, suggesting they might be "quiet" teachers.
  • Four "Teaching Styles": The authors categorize RMs into four profiles:
  • Consistent and Separating: High separation and stability (e.g., URM-LLaMa-3.1-8B).
  • Strong but Unstable: Decisive but inconsistent across prompts (e.g., Skywork-Reward-Llama-3.1-8B).
  • Broadly Discriminative but Indecisive: Spreads scores out but rarely picks a clear winner (e.g., beaver-7b-v2.0).
  • Conservative and Cautious: Stable but provides very weak signals (e.g., tulu-v2.5-13b).
  • Predicting Convergence: The "peaky" models (high SEI) tend to show much faster reward improvement in early-stage RLHF training compared to conservative models, even when their starting accuracies are similar.

Key Results

  • Leaderboard Performance: URM-LLaMa-3.1-8B emerged as the top performer on the RVB composite score, combining high separation with the best decision stability (DCI) in the group.
  • Regression Analysis: In a small-scale RLHF study using the GRPO algorithm, the authors found that accuracy alone explained only 33% of the variance in downstream reward improvement (AUC). Adding the RVB variance metrics boosted that explanatory power to 55%.
  • Semantic Robustness: The auxiliary "Eval-Semantic" test revealed that RM-Mistral-7B was the most robust to paraphrasing, while some larger models were surprisingly sensitive to minor wording changes.

Limitations and Future Work

The authors acknowledge several constraints:

  • Scale: The RLHF verification was limited to a single policy family (Qwen2.5-1.5B) and a single dataset (Anthropic HH-RLHF).
  • Scope: The current benchmark focuses on "helpfulness." Future versions need to include "harmlessness" and safety dimensions.
  • Short Horizons: The training runs were relatively short; it remains to be seen if high-variance models lead to better final convergence or just faster initial learning.
  • Generator Bias: While they tested a Qwen-based variant, the main benchmark relies on GPT-4o-generated candidates, which might subtly favor certain reward model architectures.

Summary updated

Abstract

Reward models (RMs) provide the core signal in reinforcement learning from human feedback (RLHF). However, most evaluations focus on pairwise accuracy and overlook how the separability and concentration of reward distributions shape the optimization landscape and convergence rate. To address this limitation, we present the Reward Variance Benchmark (RVB), an evaluation suite that quantifies distributional properties of RM scores. RVB introduces three variance-oriented metrics that capture complementary aspects of an RM’s signal: score distribution concentration, global pairwise separation, and cross-prompt decision-style stability. We evaluate 23 widely used RMs with a toolkit that supports reproducible analysis. On this benchmark, the variance metrics yield stable rankings and, together with accuracy, show preliminary links to downstream convergence behavior in a small-scale RLHF case study. These findings support a key insight: in addition to judging responses correctly, an effective RM in RLHF should also score them with sufficient clarity and stability to provide actionable gradients. Overall, RVB provides a first variance-centric predictive benchmark for analyzing RLHF convergence under a fixed setup, while also supporting more open-ended studies of how reward variance interacts with calibration and policy dependence.
BibTeX
@misc{li2026beyond,
  author = {Jinnan Li and Yuxing Cheng and Yue Wang and Yuan Wu and Yi Chang},
  title = {{Beyond Accuracy: Measuring Reward Variance as a Predictive Benchmark for RLHF}},
  year = {2026},
  url = {https://openreview.net/forum?id=kBoutQAl2O},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}
Figure