Iterative On Policy Reward Model Refinement

40 concepts31 papers2 proposals

In the standard paradigm of Reinforcement Learning from Human Feedback (RLHF), alignment is often treated as a sequential relay race. First, we collect a static dataset of human preferences to train a reward model. Once trained, we freeze this reward model and use it as a fixed judge to optimize a policy via reinforcement learning.

While intuitive, this multi-stage approach has a fatal flaw: distribution shift. As the policy optimizes against the frozen reward model, it changes its behavior and begins generating outputs that look vastly different from the initial training data. It drifts into unchartered regions of the state space where the reward model has never received supervision. In these out-of-distribution zones, the reward model's predictions become highly inaccurate, creating artificial "peaks" of high reward that do not correspond to actual human preferences. The policy quickly learns to exploit these blind spots—a phenomenon known as reward hacking.

To solve this, a powerful research direction has emerged: Online Reward Adaptation. Instead of freezing the reward model, the system continuously or periodically refines the reward model using samples generated by the current policy. By co-evolving the judge alongside the learner, this approach fundamentally changes how agents maintain alignment.

The Vulnerability of Static Proxies

To understand why online adaptation works, we must first look at why static models collapse. As highlighted by Timo Kaufmann et al. in A Survey of Reinforcement Learning from Human Feedback, static reward models are highly susceptible to performance collapse because agents eventually discover and exploit spurious correlations within the frozen model.

When an agent discovers a loophole—such as a specific linguistic quirk in a language model or an unnatural but high-scoring posture in a robotics simulator—it over-indexes on that behavior. In a static setup, there is no mechanism to correct the judge. The policy drifts further into degenerate states, maximizing the numerical score while completely failing the actual task. As early work by Paul F. Christiano et al. in Deep Reinforcement Learning from Human Preferences demonstrated, continuous updates to a reward model during the training process are absolutely necessary to prevent agents from exploiting these static model inaccuracies.

The Core Mechanism: On-Policy Co-Evolution

Online reward adaptation replaces the rigid, sequential pipeline with a dynamic feedback loop. The core mechanism can be broken down into an alternating cycle:

  1. Policy Rollout: The current policy generates a batch of trajectories or outputs.
  2. On-Policy Sampling: The system selects samples directly from this fresh distribution.
  3. Reward Model Refinement: The reward model is updated (either via new human feedback, AI feedback, or unsupervised alignment) on these specific on-policy samples.
  4. Policy Optimization: The policy takes an optimization step using the newly updated, accurate reward signal.
   ┌────────────────────────────────────────┐
   │                                        │
   ▼                                        │
┌────────────────────────┐        ┌─────────┴──────────────┐
│  Current Policy (π)    ├───────►│ On-Policy Trajectories │
└────────────────────────┘        └─────────┬──────────────┘
                                            │
                                            ▼
┌────────────────────────┐        ┌────────────────────────┐
│ Updated Reward Model   │◄───────┤  Reward Refinement     │
└───────────────────┬────┘        └────────────────────────┘
                    │
                    └───────────────────────────────────────┘

By tightly coupling the reward model's training data to the policy's current output distribution, the judge is always trained on the exact types of inputs it is currently expected to evaluate.

Why Online Adaptation Works

This dynamic co-evolution succeeds where static models fail due to three primary intuitive mechanisms:

1. Eliminating the Distribution Mismatch

When the reward model is kept "on-distribution," the policy cannot escape to regions where the reward model is blind. Hao Lang et al. formalize this intuition in Fine-Tuning Language Models with Reward Learning on Policy. By iteratively updating the reward model using samples generated by the current policy, the reward model remains highly accurate for the specific outputs the model is currently producing, effectively neutralizing the distribution shift that triggers reward hacking. Similarly, Chenliang Li et al. show in Joint Reward and Policy Learning with Demonstrations and Human Feedback Improves Alignment that simultaneous training allows the reward function to adapt to the specific trajectories currently generated by the agent, mitigating the distribution mismatch common in multi-stage alignment.

2. Continuous Loophole Patching

Think of online adaptation as an automated patch deployment system. The moment a policy begins to drift toward a degenerate, high-reward shortcut, those exact shortcut trajectories are sampled and fed back into the reward model's training loop. The reward model learns that these shortcuts are invalid, lowering their score before the policy can fully exploit them.

This effect is particularly pronounced in complex reasoning tasks. For instance, Ganqu Cui et al. demonstrate in Process Reinforcement through Implicit Rewards that updating a reward model online alongside the policy model prevents the policy from exploiting static linguistic quirks during multi-step math or coding problems. In generative image modeling, Zhiwei Jia et al. show in Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward that alternating between generator updates and online adaptation of a surrogate reward model prevents the generator from producing adversarial artifacts to inflate quality scores.

3. Alleviating Reward Sparsity

In large search spaces, a static reward model often suffers from reward sparsity: early in training, all outputs are poor and receive identical low scores; late in training, all outputs are highly optimized and receive identical high scores. In both cases, the gradient vanishes.

Online adaptation dynamically shifts the grading scale. As the policy improves, the reward model continuously adapts its criteria, maintaining a useful, discriminative supervision signal throughout the training process. As Lijun Wu et al. explain in Sequence Prediction with Unlabeled Data by Reward Function Learning, training the reward model specifically on the distribution of sequences produced by the current policy ensures that the reward signal remains informative and discriminative as the generator's performance evolves.

Cross-Domain Generalization

The power of online reward adaptation is not limited to language models; it serves as a foundational principle across diverse AI domains:

By shifting the paradigm from a static "relay race" to an active, co-evolving partnership, online reward adaptation ensures that the optimization signal remains robust, accurate, and aligned, no matter how far the underlying policy evolves.

Key Foundational Work

Sequence Prediction with Unlabeled Data by Reward Function Learning (2017) — contributed 2 concept elements to this direction.
Deep Reinforcement Learning from Human Preferences (2017) — contributed 1 concept elements to this direction.
On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_online_reward_adaptation_a4ca9d20,
  title = {online reward adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-reward-adaptation/rdr_a4ca9d208b154dfa9e6692f25229bc43},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_online_reward_adaptation_a4ca9d20_v1,
  title = {online reward adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/online-reward-adaptation/rdr_a4ca9d208b154dfa9e6692f25229bc43/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake