Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger, Doina Precup

Abstract

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

The Challenge of Batch Reinforcement Learning

In real-world domains like healthcare or autonomous driving, exploration is often impossible due to safety or cost constraints. We must instead rely on "batch data"—pre-collected logs of behavior. While off-policy algorithms like DDPG are theoretically capable of learning from any data, they often fail in practice when restricted to a static dataset.

The authors identify extrapolation error as the root cause. This error occurs when the Q-function is queried for state-action pairs $(s, a)$ that are not well-represented in the training data. In online RL, an agent can correct overestimations by trying the action. In a batch setting, these errors accumulate, leading to divergent value estimates and catastrophic performance.

Standard off-policy RL (DDPG) fails in batch settings due to extrapolation error, where value estimates diverge significantly from the true return.

Batch-Constrained Deep Q-Learning (BCQ)

To combat extrapolation error, the authors propose Batch-Constrained deep Q-learning (BCQ). The fundamental idea is to restrict the policy's action space to actions that are likely under the behavior policy that generated the data. By forcing the agent to stay close to the data distribution, the algorithm avoids the unreliable regions of the state-action space where extrapolation error is highest.

BCQ consists of four main components:

  • Generative Model (VAE): A Conditional Variational Auto-Encoder models the state-conditioned action distribution $P(a|s)$ of the batch, ensuring the agent only considers "plausible" actions.
  • Perturbation Model: A network that slightly adjusts actions sampled from the VAE to allow for local policy improvement beyond simple imitation.
  • Clipped Double Q-learning: Two Q-networks are used, taking the minimum of their estimates to provide a conservative value update and penalize high-variance regions.
  • Action Selection: At inference time, the agent samples multiple actions from the VAE, perturbs them, and selects the one with the highest predicted Q-value.

Experimental Evaluation

The authors evaluated BCQ on OpenAI Gym MuJoCo environments (Hopper-v1, HalfCheetah-v1, Walker2d-v1) using four types of batch data to test robustness across different data qualities:

  • Final Buffer: Transitions from a full 1-million step DDPG training run.
  • Concurrent: Data collected while the batch agent is learning simultaneously.
  • Imitation: Expert demonstrations from a single agent.
  • Imperfect Demonstrations: Expert data mixed with significant noise and random actions.

Key Results and Discussion

BCQ consistently outperformed or matched the behavioral policy and standard off-policy baselines across all tasks and data distributions. Notably, while DDPG and DQN often diverged—reaching value estimates orders of magnitude higher than the true return—BCQ maintained stable and accurate estimates.

BCQ outperforms standard RL and imitation learning baselines across various batch data distributions (Final Buffer, Concurrent, and Imitation).

In the "Imperfect Demonstrations" setting, BCQ demonstrated the ability to extract a high-performing policy even from noisy data, significantly outperforming the noisy demonstrator. This highlights BCQ's role as a bridge between imitation learning and traditional reinforcement learning.

Conclusion

BCQ represents a milestone as the first algorithm to enable effective continuous control in high-dimensional batch RL settings without any environment interaction. By explicitly constraining the policy to the data distribution, it avoids the pitfalls of extrapolation error. Future work may involve improving the generative models or incorporating more formal uncertainty quantification to further refine action selection.

Summary updated

Abstract

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.
BibTeX
@inproceedings{fujimoto2019off,
  author = {Scott Fujimoto and David Meger and Doina Precup},
  title = {{Off-Policy Deep Reinforcement Learning without Exploration}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=BybiDsWObS},
}
Figure