Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, Sergey Levine

Introduction

Offline reinforcement learning (also known as Batch RL) promises the ability to train high-performing agents from static, pre-collected datasets—such as driving logs or robot movements—without requiring further interaction with the environment. However, standard off-policy algorithms like Soft Actor-Critic (SAC) or TD3 often fail miserably in this setting.

The primary culprit is bootstrapping error. During training, an agent updates its estimate of a state's value (Q-value) based on the maximum Q-value of the next state. If the agent considers an "out-of-distribution" (OOD) action—one not present in the training dataset—the Q-function might produce a wildly inaccurate, overly optimistic value. The agent then "bootstraps" from this error, leading to a feedback loop where it pursues imaginary rewards until the policy completely diverges.

To address this, researchers introduce BEAR (Bootstrapping Error Accumulation Reduction), an algorithm that stabilizes learning by constraining the agent to only choose actions within the "support" of the training data, allowing it to outperform both standard RL and simple imitation learning.

The BEAR Algorithm

BEAR builds on the concept of distribution-constrained backups, but introduces a critical innovation in how it defines "closeness" to the training data.

Support-Set Matching vs. Distribution Matching

Previous offline RL methods, such as BCQ, attempted to make the learned policy closely match the exact distribution of the behavior policy that collected the data. BEAR argues this is too restrictive. If the data was collected by a mediocre or highly exploratory agent, copying its exact action probabilities limits the new agent's potential. Instead, BEAR simply requires the agent to stay within the range (or "support") of actions previously taken. You don't need to follow the data's exact distribution to be safe; you just need to avoid actions the Q-function knows nothing about.

Gridworld visualization of support constraints

Maximum Mean Discrepancy (MMD)

To enforce this support constraint without needing to know the exact probability density of the original data, BEAR uses Maximum Mean Discrepancy (MMD). MMD is a kernel-based distance measure between two sets of samples. By minimizing the MMD between the agent's proposed actions and the dataset's actions, BEAR ensures the agent stays within known regions of the action space. Using MMD with a limited number of samples effectively acts as a support constraint rather than a strict distribution constraint, providing the flexibility needed for the agent to improve upon the data collector.

Q-Ensemble

To further mitigate risk, BEAR uses an ensemble of $K$ Q-functions. During the Bellman backup, it takes a conservatively weighted minimum of these estimates, reducing the chance of overestimating the value of OOD actions.

Experimental Results

The researchers evaluated BEAR on several continuous control tasks from the MuJoCo physics engine (HalfCheetah, Walker2d, Hopper, Ant). They tested three specific types of offline datasets to measure robustness:

  • Random: Data collected by a completely random policy.
  • Medium: Data from a partially trained, mediocre policy (simulating imperfect human demonstrations).
  • Expert: Data from a fully optimized policy.
Performance of BEAR vs baselines on MuJoCo tasks

Key Findings

  • Robustness Across Data Types: BEAR was the only algorithm that performed consistently well across all data types (Random, Medium, and Expert).
  • Outperforming the "Teacher": On "Medium" quality data, BEAR consistently learned policies that were significantly better than the behavior policy that generated the data. For example, in the Walker2d task, BEAR achieved a score of ~3000, while the underlying data averaged only ~1000.
  • Beating Strict Distribution Matching: While previous state-of-the-art methods like BCQ performed well on expert data, they struggled on random or mediocre data. Their constraints were too aggressive, essentially forcing the agent into behavioral cloning rather than allowing it to optimize within the support set.

Limitations and Future Work

While BEAR represents a significant step forward for offline RL, challenges remain:

  • Performance Degradation: In very long training runs, performance can sometimes still degrade. Developing an "early stopping" condition for RL—similar to validation sets in supervised learning—is a critical next step.
  • High-Dimensional Action Spaces: While BEAR works well on standard MuJoCo tasks, scaling support-set matching to higher-dimensional or more complex action spaces remains an open challenge.
  • State Distribution Shift: BEAR focuses strictly on action constraints. However, the agent might still visit states that are OOD. Future work could explore directly constraining the state distribution to further stabilize offline learning.

Summary updated

Abstract

Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only limited progress without collecting additional on-policy data. As a step towards more robust off-policy algorithms, we study the setting where the off-policy experience is fixed and there is no further interaction with the environment. We identify \emph{bootstrapping error} as a key source of instability in current methods. Bootstrapping error is due to bootstrapping from actions that lie outside of the training data distribution, and it accumulates via the Bellman backup operator. We theoretically analyze bootstrapping error, and demonstrate how carefully constraining action selection in the backup can mitigate it. Based on our analysis, we propose a practical algorithm, bootstrapping error accumulation reduction (BEAR). We demonstrate that BEAR is able to learn robustly from different off-policy distributions, including random data and suboptimal demonstrations, on a range of continuous control tasks.
BibTeX
@inproceedings{kumar2019stabilizing,
  author = {Aviral Kumar and Justin Fu and Matthew Soh and George Tucker and Sergey Levine},
  title = {{Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=2DVCDRE02h},
}
Figure