Reinforcement learning (RL) has historically been divided into two camps: on-policy methods, which are stable but "wasteful," and off-policy methods, which are efficient but "dangerous."
On-policy algorithms like Proximal Policy Optimization (PPO) only learn from data generated by the current version of the agent. This ensures the gradient updates are unbiased and stable, but it requires discarding expensive data after a single use. Conversely, off-policy methods like Deep Q-Networks (DQN) use an experience replay buffer to reuse old data. While this is far more sample-efficient, it often leads to the "deadly triad"—a combination of function approximation, bootstrapping, and off-policy training that can cause the model's value estimates to diverge or collapse.
A major research direction has emerged to bridge this gap, creating hybrid algorithms that combine the reliable convergence of on-policy gradients with the high throughput of off-policy data reuse.
The Stability-Efficiency Trade-off
The core tension in RL is between the variance of the gradient and the bias of the value estimate. As noted in How to Choose a Reinforcement-Learning Algorithm by Fabian Bongratz, practitioners often face a binary choice: the stability of first-order on-policy methods or the sample efficiency of off-policy data reuse.
In complex domains like continuous control or quantum state preparation, on-policy methods are often preferred because they reach higher final fidelity, even if they take longer to get there. For instance, Shengyong Li found in Robust quantum control using reinforcement learning from demonstration that while off-policy methods learn faster, they struggle to achieve the extreme precision required for high-fidelity operations.
Reducing Variance with Off-Policy Control Variates
One way to bridge the gap is to use an off-policy model not to replace the on-policy gradient, but to "assist" it. Shixiang Gu, Sergey Levine, and colleagues introduced Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic, which uses a Taylor expansion of an off-policy critic as a control variate.
The intuition is simple: if you have a noisy estimate (the on-policy gradient) and a biased but stable estimate (the off-policy critic), you can subtract the critic's "guess" from the gradient and then add back its analytical mean. This reduces the variance of the update without introducing the bias that causes off-policy methods to diverge. Similarly, Interpolated Policy Gradient blends these two signals using a tunable parameter, allowing the agent to lean on off-policy data early in training and shift toward on-policy stability as it nears an optimal solution.
Trust Regions and Experience Replay
Another approach focuses on making experience replay safe for policy gradients. Standard policy gradients assume the data comes from the current policy; if the data is "stale" (from an older version of the model), the gradient points in the wrong direction.
To fix this, researchers have developed advanced correction mechanisms:
- Importance Sampling: Ziyu Wang and the team at DeepMind developed ACER (Actor-Critic with Experience Replay), which uses truncated importance sampling and a trust-region optimization to ensure that updates from old data don't push the policy too far from its current state.
- V-trace and LASER: Simon Schmitt introduced LASER, which mixes a small amount of fresh on-policy data with a massive buffer of off-policy data, using the V-trace algorithm to correct for the distribution shift.
- Early Stopping: Instead of complex clipping, Vitaly Kurin and others proposed ESPO, which simply stops the optimization process the moment the updated policy deviates too far from the data-collection policy.
Scaling to Large Language Models
The need for this hybrid approach has become critical in the era of Large Language Models (LLMs). Training an LLM with RL (like RLHF) is computationally expensive because generating "rollouts" (text responses) is slow.
Recent work like M2PO (Second-Moment Trust Policy Optimization) by Haizhong Zheng and Beidi Chen demonstrates that models can effectively train on "stale" data—up to 256 updates old—by using second-moment constraints to prevent model collapse. Similarly, Batch Adaptation Policy Optimization (BAPO) by Xu Wan shows that maintaining a buffer of successful historical reasoning paths allows LLMs to "unlock" complex problems that they would otherwise forget how to solve if they only relied on fresh, on-policy rollouts.
By treating the off-policy buffer as a source of "advice" or a "baseline" rather than the sole source of truth, these hybrid methods allow agents—from robots to reasoning models—to learn faster without the catastrophic instability that once defined off-policy reinforcement learning.