Regularized Behavior Value Estimation

Çaglar Gülçehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew W. Hoffman, Razvan Pascanu, Nando de FreitasarXiv 2021 · arXiv

Abstract

Regularized Behavior Value Estimation (R-BVE) addresses the challenge of value over-estimation in offline reinforcement learning. By decoupling policy evaluation from policy improvement—learning the behavior policy's value during training and performing a single improvement step at deployment—R-BVE avoids the cumulative errors of standard Q-learning. Combined with ranking regularization and success filtering, R-BVE achieves state-of-the-art performance on Atari benchmarks and demonstrates superior robustness in low-data and noisy environments.

Introduction

In offline reinforcement learning (RL), agents learn from fixed datasets of logged experiences without environment interaction. The primary challenge is the "vicious loop" of value over-estimation. Standard RL algorithms use a max operator (e.g., Q-learning) to identify optimal actions. In offline settings, agents frequently encounter out-of-distribution (OOD) state-action pairs. Function approximation errors can cause neural networks to assign arbitrarily high values to these OOD actions. Without real-world interaction to correct these errors, they bootstrap and amplify, leading to training divergence and "delusional" agent behavior.

Methods

R-BVE proposes a fundamental shift in the offline RL training paradigm to break this cycle:

  1. Behavior Value Estimation (BVE): Instead of iteratively learning the optimal value function ($Q^*$) during training, R-BVE estimates the value of the behavior policy ($Q^{\pi_B}$) that generated the data. This removes the unstable max operator from the training loop. Policy improvement is deferred to a single greedy step performed only at deployment.
  2. Ranking Regularization (R-): To safeguard the value function, a max-margin ranking loss is added. This loss explicitly ensures that actions found in successful trajectories are valued higher than unobserved or unsuccessful actions, preventing OOD actions from dominating.
  3. Success Filtering: Transitions are re-weighted based on the total return of their parent trajectory, ensuring the ranking loss promotes high-quality behavior rather than just frequent behavior.
Comparison of standard offline RL training versus the R-BVE approach.

Key Insights

The authors demonstrate that the repeated application of the max operator during training is the primary driver of instability in offline RL. By deferring policy improvement to deployment, R-BVE avoids the cumulative error propagation seen in standard Q-learning. Theoretically and empirically, a single step of policy improvement is shown to be highly effective, capturing most potential performance gains in complex tasks. Furthermore, ranking actions is inherently more robust than attempting to regress exact values for unobserved states.

Key Results

R-BVE was evaluated across the RL Unplugged Atari suite, bsuite diagnostic tasks, and complex 3D environments in DeepMind Lab.

  • Atari State-of-the-Art: R-BVE achieved a median normalized score of 109.1% across the Atari suite, outperforming existing state-of-the-art methods such as CQL (98.9%) and REM (104.7%).
  • Low-Data Robustness: In experiments where datasets were sub-sampled to just 1% of their original size, R-BVE maintained stable value estimates, whereas standard DQN-based methods suffered from massive over-estimation.
  • DeepMind Lab: R-BVE consistently outperformed R2D2 and CQL in partially observable 3D tasks, suggesting better handling of data coverage issues in high-dimensional environments.
Atari suite performance comparison showing R-BVE achieving state-of-the-art results.

Conclusion and Future Work

Currently, the ranking loss formulation is optimized for discrete action spaces. While extensions to continuous control are possible, they remain to be experimentally validated. Additionally, R-BVE's effectiveness depends on the presence of some successful trajectories in the behavior data; if the dataset contains only failures, the success filter has no positive behavior to promote. Future research will explore applications in continuous control and real-world robotics where data efficiency and safety are critical.

Summary updated

Abstract

Offline reinforcement learning restricts the learning process to rely only on logged-data without access to an environment. While this enables real-world applications, it also poses unique challenges. One important challenge is dealing with errors caused by the overestimation of values for state-action pairs not well-covered by the training data. Due to bootstrapping, these errors get amplified during training and can lead to divergence, thereby crippling learning. To overcome this challenge, we introduce Regularized Behavior Value Estimation (R-BVE). Unlike most approaches, which use policy improvement during training, R-BVE estimates the value of the behavior policy during training and only performs policy improvement at deployment time. Further, R-BVE uses a ranking regularisation term that favours actions in the dataset that lead to successful outcomes. We provide ample empirical evidence of R-BVE's effectiveness, including state-of-the-art performance on the RL Unplugged ATARI dataset. We also test R-BVE on new datasets, from bsuite and a challenging DeepMind Lab task, and show that R-BVE outperforms other state-of-the-art discrete control offline RL methods.
BibTeX
@misc{gulcehre2021regularized,
  author = {Çaglar Gülçehre and Sergio Gómez Colmenarejo and Ziyu Wang and Jakub Sygnowski and Thomas Paine and Konrad Zolna and Yutian Chen and Matthew W. Hoffman and Razvan Pascanu and Nando de Freitas},
  title = {{Regularized Behavior Value Estimation}},
  year = {2021},
  eprint = {2103.09575},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.09575},
}
Figure