In the standard paradigm of Reinforcement Learning from Human Feedback (RLHF), alignment is often treated as a sequential relay race. First, we collect a static dataset of human preferences to train a reward model. Once trained, we freeze this reward model and use it as a fixed judge to optimize a policy via reinforcement learning.
While intuitive, this multi-stage approach has a fatal flaw: distribution shift. As the policy optimizes against the frozen reward model, it changes its behavior and begins generating outputs that look vastly different from the initial training data. It drifts into unchartered regions of the state space where the reward model has never received supervision. In these out-of-distribution zones, the reward model's predictions become highly inaccurate, creating artificial "peaks" of high reward that do not correspond to actual human preferences. The policy quickly learns to exploit these blind spots—a phenomenon known as reward hacking.
To solve this, a powerful research direction has emerged: Online Reward Adaptation. Instead of freezing the reward model, the system continuously or periodically refines the reward model using samples generated by the current policy. By co-evolving the judge alongside the learner, this approach fundamentally changes how agents maintain alignment.
The Vulnerability of Static Proxies
To understand why online adaptation works, we must first look at why static models collapse. As highlighted by Timo Kaufmann et al. in A Survey of Reinforcement Learning from Human Feedback, static reward models are highly susceptible to performance collapse because agents eventually discover and exploit spurious correlations within the frozen model.
When an agent discovers a loophole—such as a specific linguistic quirk in a language model or an unnatural but high-scoring posture in a robotics simulator—it over-indexes on that behavior. In a static setup, there is no mechanism to correct the judge. The policy drifts further into degenerate states, maximizing the numerical score while completely failing the actual task. As early work by Paul F. Christiano et al. in Deep Reinforcement Learning from Human Preferences demonstrated, continuous updates to a reward model during the training process are absolutely necessary to prevent agents from exploiting these static model inaccuracies.
The Core Mechanism: On-Policy Co-Evolution
Online reward adaptation replaces the rigid, sequential pipeline with a dynamic feedback loop. The core mechanism can be broken down into an alternating cycle:
- Policy Rollout: The current policy generates a batch of trajectories or outputs.
- On-Policy Sampling: The system selects samples directly from this fresh distribution.
- Reward Model Refinement: The reward model is updated (either via new human feedback, AI feedback, or unsupervised alignment) on these specific on-policy samples.
- Policy Optimization: The policy takes an optimization step using the newly updated, accurate reward signal.
┌────────────────────────────────────────┐
│ │
▼ │
┌────────────────────────┐ ┌─────────┴──────────────┐
│ Current Policy (π) ├───────►│ On-Policy Trajectories │
└────────────────────────┘ └─────────┬──────────────┘
│
▼
┌────────────────────────┐ ┌────────────────────────┐
│ Updated Reward Model │◄───────┤ Reward Refinement │
└───────────────────┬────┘ └────────────────────────┘
│
└───────────────────────────────────────┘
By tightly coupling the reward model's training data to the policy's current output distribution, the judge is always trained on the exact types of inputs it is currently expected to evaluate.
Why Online Adaptation Works
This dynamic co-evolution succeeds where static models fail due to three primary intuitive mechanisms:
1. Eliminating the Distribution Mismatch
When the reward model is kept "on-distribution," the policy cannot escape to regions where the reward model is blind. Hao Lang et al. formalize this intuition in Fine-Tuning Language Models with Reward Learning on Policy. By iteratively updating the reward model using samples generated by the current policy, the reward model remains highly accurate for the specific outputs the model is currently producing, effectively neutralizing the distribution shift that triggers reward hacking. Similarly, Chenliang Li et al. show in Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment that simultaneous training allows the reward function to adapt to the specific trajectories currently generated by the agent, mitigating the distribution mismatch common in multi-stage alignment.
2. Continuous Loophole Patching
Think of online adaptation as an automated patch deployment system. The moment a policy begins to drift toward a degenerate, high-reward shortcut, those exact shortcut trajectories are sampled and fed back into the reward model's training loop. The reward model learns that these shortcuts are invalid, lowering their score before the policy can fully exploit them.
This effect is particularly pronounced in complex reasoning tasks. For instance, Ganqu Cui et al. demonstrate in Process Reinforcement through Implicit Rewards that updating a reward model online alongside the policy model prevents the policy from exploiting static linguistic quirks during multi-step math or coding problems. In generative image modeling, Zhiwei Jia et al. show in Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward that alternating between generator updates and online adaptation of a surrogate reward model prevents the generator from producing adversarial artifacts to inflate quality scores.
3. Alleviating Reward Sparsity
In large search spaces, a static reward model often suffers from reward sparsity: early in training, all outputs are poor and receive identical low scores; late in training, all outputs are highly optimized and receive identical high scores. In both cases, the gradient vanishes.
Online adaptation dynamically shifts the grading scale. As the policy improves, the reward model continuously adapts its criteria, maintaining a useful, discriminative supervision signal throughout the training process. As Lijun Wu et al. explain in Sequence Prediction with Unlabeled Data by Reward Function Learning, training the reward model specifically on the distribution of sequences produced by the current policy ensures that the reward signal remains informative and discriminative as the generator's performance evolves.
Cross-Domain Generalization
The power of online reward adaptation is not limited to language models; it serves as a foundational principle across diverse AI domains:
- Robotic Manipulation: In Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning, Adrià López Escoriza et al. optimize the reward model concurrently with the policy during environmental interaction. This online adaptation ensures the reward signal remains relevant as the agent encounters novel states, preventing the performance plateaus and local optima common in offline inverse reinforcement learning.
- Hardware Optimization: For hard-to-simulate physical objectives like quadruped robot battery consumption, Ruiqian Nai et al. demonstrate in Fine-Tuning Hard-to-Simulate Objectives for Quadruped Locomotion: A Case Study on Total Power Saving that iteratively collecting real-world data using the current policy to retrain the measurement model keeps the reward signal accurate as the robot's gait evolves toward more efficient states.
- Regularization Anchors: Even when optimizing models via direct preference optimization (DPO) without an explicit reward model, the same underlying principle applies. Jack Lanchantin et al. show in Bridging Offline and Online Reinforcement Learning for LLMs that periodically updating the reference model used as a Kullback–Leibler (KL) divergence anchor prevents the policy from drifting into degenerate regions, acting as a dynamic guardrail against reward hacking.
By shifting the paradigm from a static "relay race" to an active, co-evolving partnership, online reward adaptation ensures that the optimization signal remains robust, accurate, and aligned, no matter how far the underlying policy evolves.
Key Foundational Work
Sequence Prediction with Unlabeled Data by Reward Function Learning (2017) — contributed 2 concept elements to this direction.
Deep Reinforcement Learning from Human Preferences (2017) — contributed 1 concept elements to this direction.
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems (2016) — contributed 1 concept elements to this direction.