Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger, Doina Precup

TL;DR

Standard off-policy reinforcement learning (RL) algorithms like DQN and DDPG fail miserably when trained on fixed datasets without the ability to explore. This paper identifies "extrapolation error" as the culprit and introduces Batch-Constrained deep Q-learning (BCQ), the first continuous control algorithm that successfully learns from arbitrary, fixed batches of data by forcing the agent to stay close to the actions seen in the dataset.

The Problem: Extrapolation Error in Batch RL

In an ideal world, we could take a massive pile of historical data—such as logs from a robot, medical records, or human driving data—and train an RL agent to perform better than the original actor. This paradigm is known as Batch Reinforcement Learning.

However, the authors show that standard "off-policy" algorithms, which theoretically should learn from any data, actually fail when they cannot interact with the environment to collect new samples. The root cause is extrapolation error. When an agent evaluates a state-action pair that isn't in the dataset, the Q-network often provides a wildly inaccurate, over-optimistic value. Because RL agents are designed to maximize the Q-value, they gravitate toward these "hallucinated" high-value actions, leading to a total collapse in performance.

Extrapolation error in standard off-policy RL

As seen above, standard off-policy DDPG often performs significantly worse than the behavioral agent that collected the data, with its value estimates diverging wildly from the true value.

Methods: Batch-Constrained Deep Q-Learning (BCQ)

To solve this, the authors propose BCQ. The core philosophy is simple: only consider actions that are likely to have been generated by the behavior policy that created the dataset.

Key components include:

  • Conditional VAE (Variational Auto-Encoder): BCQ trains a generative model to capture the distribution of actions found in the batch for any given state. When acting, the agent samples candidate actions from this VAE.
  • Perturbation Model: To allow the agent to improve upon the historical data rather than just mimicking it, a small perturbation network adds a bounded adjustment to the sampled actions, enabling local optimization.
  • Modified Clipped Double Q-learning: To combat overestimation bias, BCQ uses two Q-networks and takes a weighted minimum of their estimates. This heavily penalizes uncertainty in regions where data is sparse.
  • Policy Selection: The final policy samples multiple actions from the VAE, perturbs them, and selects the one that maximizes the estimated Q-value.

Experimental Setup

The algorithm was evaluated using OpenAI Gym’s MuJoCo environments (Hopper-v1, HalfCheetah-v1, and Walker2d-v1). The authors created distinct types of batch datasets to test different scenarios:

  1. Final Buffer: All transitions from an agent trained for 1 million steps with high exploration noise.
  2. Concurrent: A dataset where a behavioral agent and the off-policy agent learn simultaneously from the same growing buffer.
  3. Imitation: 1 million transitions from a pre-trained expert agent.
  4. Imperfect Demonstrations: A noisy dataset mixing expert and random actions.

Key Results and Insights

The paper highlights a critical "Off-Policy Illusion": most modern off-policy algorithms are actually "growing batch" learners. They rely heavily on their current policy remaining somewhat correlated with the data in their replay buffer. When that correlation breaks in pure Batch RL, they fail.

BCQ performance compared to baselines
  • Stability and Performance: BCQ showed remarkably stable value estimates, whereas DDPG and DQN value estimates often diverged. In the "Final Buffer" and "Concurrent" tasks, BCQ consistently outperformed the behavioral policy and crushed standard DDPG/DQN.
  • Robustness: In the "Imperfect Demonstrations" task, BCQ successfully disentangled good actions from noisy ones, outperforming Behavioral Cloning (which just mimics the noise) and DDPG (which diverges).
  • Efficiency: BCQ reached high performance in significantly fewer iterations than many state-of-the-art online RL algorithms.

Limitations and Future Work

  • Deterministic Assumption: Much of the theoretical proof relies on deterministic MDPs. Extending the theory to highly stochastic environments remains an open challenge.
  • VAE Dependency: BCQ's performance is bottlenecked by the quality of the VAE. If the generative model fails to capture the data distribution accurately, the agent's action space will be poorly constrained.
  • Hyperparameter Tuning: The algorithm introduces a weighting parameter to balance the minimum and maximum of the twin Q-networks. While a single value worked across MuJoCo tasks, it might require careful tuning in more complex real-world domains.

Summary updated

Abstract

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection....
BibTeX
@inproceedings{fujimoto2019off,
  author = {Scott Fujimoto and David Meger and Doina Precup},
  title = {{Off-Policy Deep Reinforcement Learning without Exploration}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BybiDsWObS},
}
Figure