Online Reward Modeling for Robust Alignment

This direction is split into other directions: online reward adaptation. Its last version stays readable and citable here.

Version 1 of 1132 papers166 conceptsfrozen 2026-10-04

In the standard Reinforcement Learning from Human Feedback (RLHF) pipeline, alignment is treated as a linear, two-stage process: first, we train a static reward model on a fixed dataset of human preferences; second, we optimize a policy against that fixed reward model.

However, this "relay race" approach has a fundamental flaw. As the policy improves, it begins to generate outputs that are far outside the distribution of the initial preference data. In these uncharted regions, the static reward model becomes an unreliable judge, often leading to reward hacking—where the policy exploits the reward model's blind spots to achieve high scores without actually improving in quality.

To solve this, a growing body of research proposes Online Reward Modeling: a paradigm where the reward model and the policy co-evolve, ensuring the "judge" remains as sophisticated as the "student" it evaluates.

The Distribution Shift Problem

The core intuition behind online reward modeling is that a reward model is only accurate for the types of data it was trained on. Hao Lang and colleagues in Fine-Tuning Language Models with Reward Learning on Policy argue that static reward models inevitably suffer from distribution shift. As the policy model evolves, it produces "on-policy" samples that the "off-policy" reward model cannot accurately rank.

This shift is what triggers overoptimization. As Thomas Coste notes in Reward Model Ensembles Help Mitigate Overoptimization, the model's performance according to the proxy reward keeps rising while its actual quality (the "gold" standard) plateaus or crashes. By updating the reward model using the policy's own current trajectories, researchers can "anchor" the reward landscape to the agent's actual capabilities.

Co-evolutionary Training Mechanisms

Rather than a one-time handoff, online reward modeling treats alignment as a continuous feedback loop. Several mechanisms have emerged to keep the reward signal synchronized with the policy:

Mitigating Reward Hacking in Reasoning

The stakes for online reward modeling are particularly high in reasoning tasks. In Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning, Jie Cheng demonstrates that static reward models often fail to provide fine-grained credit assignment, leading models to "over-think" or use repetitive templates to hack rewards.

To combat this, Hu Wang and Congbo Ma proposed KRPO (Kalman Filter Enhanced GRPO), which uses dynamic filtering to track the evolving reward distribution. This allows the model to distinguish between genuine reasoning improvements and statistical noise in the reward signal. Similarly, the InternLM2 technical report by Chengqi Lyu highlights that "COOL RLHF" relies on multiple rounds of fresh feedback to prevent the policy from exploiting static flaws in the initial reward function.

Efficiency and Scalability

A major hurdle for online reward modeling is the computational cost of constantly querying a "gold" reward (like a human or a massive teacher model).

By treating the reward model as a dynamic, evolving entity rather than a static artifact, this research direction moves AI alignment away from "one-shot" training toward a more robust, interactive, and stable co-evolutionary process.

Papers in this version

Showing 60 of 132 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_online_reward_modeling_for_robust_alignm_98f46001,
  title = {Online Reward Modeling for Robust Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-reward-modeling-for-robust-alignment/rdr_98f46001fb3344e5894c32c9e84a9a3f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_online_reward_modeling_for_robust_alignm_98f46001_v1,
  title = {Online Reward Modeling for Robust Alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-reward-modeling-for-robust-alignment/rdr_98f46001fb3344e5894c32c9e84a9a3f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}