The Reward-Utility Gap in Reinforcement Learning Alignment

This direction is merged into another direction: reward hacking (proxy objective exploitation). Its last version stays readable and citable here.

Version 1 of 132 papers39 conceptsfrozen 2026-10-04

In the standard pipeline for aligning Large Language Models (LLMs), the Reward Model (RM) acts as a digital compass. During Reinforcement Learning from Human Feedback (RLHF), the RM tells the policy which responses are "good" and which are "bad," guiding the model toward human-preferred behavior.

However, a growing body of research suggests that our current compasses are broken. Models that achieve state-of-the-art scores on reward benchmarks often fail to produce better language models when used for training. This phenomenon, known as the reward-utility gap, reveals a fundamental decoupling between how we evaluate reward models and how they actually perform in the wild.

The Accuracy Paradox

The industry standard for evaluating an RM is "pairwise accuracy": given two responses, A and B, can the RM correctly identify which one a human preferred? While this seems intuitive, Xueru Wen and colleagues in Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? demonstrate that this metric is a surprisingly weak predictor of downstream performance.

Two reward models with identical accuracy can lead to wildly different alignment outcomes. This is because pairwise accuracy is a static metric—it measures a model's ability to rank a fixed set of responses. In contrast, RLHF is a dynamic process where the policy actively tries to find "hacks" to maximize the reward. As Evan Frick and his team show in How to Evaluate Reward Models for RLHF, for high-performing models, performance on static benchmarks can even correlate negatively with real-world human preference scores.

Why the Gap Exists: Style, Substance, and Signal

Three primary mechanisms explain why high-accuracy reward models fail to deliver high-utility policies.

1. Style Outweighs Substance

Reward models are easily "hacked" by superficial features. Benjamin Feuer and others introduced SOS-BENCH to show that LLM-judges often prioritize "vibes"—such as verbosity and politeness—over factual accuracy or safety. Similarly, Yantao Liu found in RM-Bench that state-of-the-art RMs frequently perform worse than random guessing when a factually incorrect answer is presented in a more appealing format.

2. Reward Overoptimization (Goodhart’s Law)

When a measure becomes a target, it ceases to be a good measure. In RLHF, the policy model explores the RM's "blind spots." Dongha Lee argues in Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization that RMs with high static accuracy may be more susceptible to this "Regressional Goodhart effect," where the policy achieves a high reward by generating nonsensical or biased text that the RM happens to like.

3. The Signal Strength Problem

A reward model must not only be right; it must be "loud" enough to teach. Noam Razin explains in What Makes a Reward Model a Good Teacher? that reward variance is a critical bottleneck. If an RM correctly ranks two responses but gives them nearly identical scores (a "flat" landscape), the policy model won't receive enough signal to update its parameters. An accurate but "quiet" RM is a useless teacher.

Toward Predictive Benchmarking

To bridge this gap, researchers are moving away from simple pairwise accuracy toward metrics that simulate the actual optimization process.

  • Best-of-N (BoN) Evaluation: Instead of ranking two fixed options, Saumya Malik and the team behind RewardBench 2 use a "Best-of-4" format. This tests if an RM can pick the single best response from a larger pool, which more closely mirrors how RMs are used in production.
  • Preference Proxy Evaluations (PPE): Evan Frick introduced PPE to identify specific reward model metrics that achieve high Pearson correlation (up to 77%) with final LLM performance, allowing developers to predict success before starting expensive training runs.
  • Evolving Rubrics: For complex tasks like deep research, Akari Asai and Rulin Shao developed Reinforcement Learning with Evolving Rubrics (RLER) in DR Tulu. This method acknowledges that the "correct" reward signal must change as the model becomes more sophisticated, preventing the reward-utility gap from widening during long training runs.

The shift in the field is clear: we are moving from evaluating reward models as static classifiers to evaluating them as optimization targets. Understanding why a model is "accurate" is no longer enough; we must understand if its accuracy is robust enough to survive the pressure of reinforcement learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_reward_utility_gap_in_reinforcement__c006cb49,
  title = {The Reward-Utility Gap in Reinforcement Learning Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-reward-utility-gap-in-reinforcement-learning-alignment/rdr_c006cb49e7cc4213aa0889bef3fce806},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_reward_utility_gap_in_reinforcement__c006cb49_v1,
  title = {The Reward-Utility Gap in Reinforcement Learning Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-reward-utility-gap-in-reinforcement-learning-alignment/rdr_c006cb49e7cc4213aa0889bef3fce806/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}