In the standard Reinforcement Learning from Human Feedback (RLHF) pipeline, alignment is treated as a linear, two-stage process: first, we train a static reward model on a fixed dataset of human preferences; second, we optimize a policy against that fixed reward model.
However, this "relay race" approach has a fundamental flaw. As the policy improves, it begins to generate outputs that are far outside the distribution of the initial preference data. In these uncharted regions, the static reward model becomes an unreliable judge, often leading to reward hacking—where the policy exploits the reward model's blind spots to achieve high scores without actually improving in quality.
To solve this, a growing body of research proposes Online Reward Modeling: a paradigm where the reward model and the policy co-evolve, ensuring the "judge" remains as sophisticated as the "student" it evaluates.
The Distribution Shift Problem
The core intuition behind online reward modeling is that a reward model is only accurate for the types of data it was trained on. Hao Lang and colleagues in Fine-Tuning Language Models with Reward Learning on Policy argue that static reward models inevitably suffer from distribution shift. As the policy model evolves, it produces "on-policy" samples that the "off-policy" reward model cannot accurately rank.
This shift is what triggers overoptimization. As Thomas Coste notes in Reward Model Ensembles Help Mitigate Overoptimization, the model's performance according to the proxy reward keeps rising while its actual quality (the "gold" standard) plateaus or crashes. By updating the reward model using the policy's own current trajectories, researchers can "anchor" the reward landscape to the agent's actual capabilities.
Co-evolutionary Training Mechanisms
Rather than a one-time handoff, online reward modeling treats alignment as a continuous feedback loop. Several mechanisms have emerged to keep the reward signal synchronized with the policy:
- Joint Optimization: Frameworks like AIHF, introduced by Chenliang Li in Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment, train the reward model and policy simultaneously. This ensures the reward function adapts in real-time to the specific trajectories the agent is currently exploring.
- Adversarial Refinement: In Adversarial Preference Optimization, Pengyu Cheng and others update the reward model to specifically differentiate between high-quality reference data and the generator's latest, most "tricky" outputs. This adversarial pressure prevents the generator from finding easy shortcuts.
- Experience Relabeling: For off-policy learning, Kimin Lee introduced PEBBLE (PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training), which periodically relabels all historical transitions in a replay buffer. This ensures that even old experiences are evaluated by the most up-to-date version of the reward model.
Mitigating Reward Hacking in Reasoning
The stakes for online reward modeling are particularly high in reasoning tasks. In Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning, Jie Cheng demonstrates that static reward models often fail to provide fine-grained credit assignment, leading models to "over-think" or use repetitive templates to hack rewards.
To combat this, Hu Wang and Congbo Ma proposed KRPO (Kalman Filter Enhanced GRPO), which uses dynamic filtering to track the evolving reward distribution. This allows the model to distinguish between genuine reasoning improvements and statistical noise in the reward signal. Similarly, the InternLM2 technical report by Chengqi Lyu highlights that "COOL RLHF" relies on multiple rounds of fresh feedback to prevent the policy from exploiting static flaws in the initial reward function.
Efficiency and Scalability
A major hurdle for online reward modeling is the computational cost of constantly querying a "gold" reward (like a human or a massive teacher model).
- Active Learning: André Eberhard and colleagues developed ACRL (Actively Learning Costly Reward Functions for Reinforcement Learning), which uses a fast neural network proxy that is only updated when the agent enters high-uncertainty regions of the state space.
- Online Draft Learning: To accelerate training, Jisheng Dang introduced FastGRPO (FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning), which keeps a small "draft" model in sync with the main model's evolving distribution, allowing for faster speculative decoding during the RL loop.
- Latent Rewards: In the domain of video generation, Liyu Zhang proposed VELR (VELR: Efficient Video Reward Feedback via Ensemble Latent Reward Models), which moves reward prediction into the latent space to slash memory usage while maintaining online alignment.
By treating the reward model as a dynamic, evolving entity rather than a static artifact, this research direction moves AI alignment away from "one-shot" training toward a more robust, interactive, and stable co-evolutionary process.