Abstract
Regularized Behavior Value Estimation (R-BVE) addresses the challenge of value over-estimation in offline reinforcement learning. By decoupling policy evaluation from policy improvement—learning the behavior policy's value during training and performing a single improvement step at deployment—R-BVE avoids the cumulative errors of standard Q-learning. Combined with ranking regularization and success filtering, R-BVE achieves state-of-the-art performance on Atari benchmarks and demonstrates superior robustness in low-data and noisy environments.
Introduction
In offline reinforcement learning (RL), agents learn from fixed datasets of logged experiences without environment interaction. The primary challenge is the "vicious loop" of value over-estimation. Standard RL algorithms use a max operator (e.g., Q-learning) to identify optimal actions. In offline settings, agents frequently encounter out-of-distribution (OOD) state-action pairs. Function approximation errors can cause neural networks to assign arbitrarily high values to these OOD actions. Without real-world interaction to correct these errors, they bootstrap and amplify, leading to training divergence and "delusional" agent behavior.
Methods
R-BVE proposes a fundamental shift in the offline RL training paradigm to break this cycle:
- Behavior Value Estimation (BVE): Instead of iteratively learning the optimal value function ($Q^*$) during training, R-BVE estimates the value of the behavior policy ($Q^{\pi_B}$) that generated the data. This removes the unstable
maxoperator from the training loop. Policy improvement is deferred to a single greedy step performed only at deployment. - Ranking Regularization (R-): To safeguard the value function, a max-margin ranking loss is added. This loss explicitly ensures that actions found in successful trajectories are valued higher than unobserved or unsuccessful actions, preventing OOD actions from dominating.
- Success Filtering: Transitions are re-weighted based on the total return of their parent trajectory, ensuring the ranking loss promotes high-quality behavior rather than just frequent behavior.
Key Insights
The authors demonstrate that the repeated application of the max operator during training is the primary driver of instability in offline RL. By deferring policy improvement to deployment, R-BVE avoids the cumulative error propagation seen in standard Q-learning. Theoretically and empirically, a single step of policy improvement is shown to be highly effective, capturing most potential performance gains in complex tasks. Furthermore, ranking actions is inherently more robust than attempting to regress exact values for unobserved states.
Key Results
R-BVE was evaluated across the RL Unplugged Atari suite, bsuite diagnostic tasks, and complex 3D environments in DeepMind Lab.
- Atari State-of-the-Art: R-BVE achieved a median normalized score of 109.1% across the Atari suite, outperforming existing state-of-the-art methods such as CQL (98.9%) and REM (104.7%).
- Low-Data Robustness: In experiments where datasets were sub-sampled to just 1% of their original size, R-BVE maintained stable value estimates, whereas standard DQN-based methods suffered from massive over-estimation.
- DeepMind Lab: R-BVE consistently outperformed R2D2 and CQL in partially observable 3D tasks, suggesting better handling of data coverage issues in high-dimensional environments.
Conclusion and Future Work
Currently, the ranking loss formulation is optimized for discrete action spaces. While extensions to continuous control are possible, they remain to be experimentally validated. Additionally, R-BVE's effectiveness depends on the presence of some successful trajectories in the behavior data; if the dataset contains only failures, the success filter has no positive behavior to promote. Future research will explore applications in continuous control and real-world robotics where data efficiency and safety are critical.
Summary updated