RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning

Çaglar Gülçehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez Colmenarejo, Konrad Zolna, Rishabh Agarwal, Josh Merel, Daniel J. Mankowitz, Cosmin Paduraru, Gabriel Dulac-Arnold, Jerry Li, Mohammad Norouzi, Matthew Hoffman, Nicolas Heess, Nando de FreitasNeurIPS 2020 · OpenReview

Overview

"RL Unplugged" introduces a comprehensive suite of benchmarks and datasets for offline reinforcement learning (RL) to address the field's reproducibility crisis. By standardizing diverse environments—from Atari games to complex robotic locomotion—and establishing strict evaluation protocols, the authors provide a robust foundation for developing RL algorithms that learn from fixed datasets without requiring costly or dangerous real-world exploration.

The Challenge of Real-World RL

While Reinforcement Learning has achieved massive success in simulated environments (like Go or StarCraft), its adoption in the real world remains limited. Real-world systems like power plants, healthcare systems, or autonomous vehicles cannot safely or cheaply support the millions of trial-and-error interactions required by standard "online" RL.

Offline RL solves this by training agents entirely on previously logged datasets. However, offline RL research is currently plagued by a reproducibility crisis. Algorithms are often evaluated on non-standardized datasets, and more problematically, researchers frequently tune their offline algorithms using online interactions with a simulator (to see which hyperparameters work best). This defeats the purpose of offline RL, giving an overly optimistic view of how these algorithms would perform in the real world where such tuning is impossible. This paper addresses this gap by providing standardized datasets, unified APIs, and strict evaluation protocols that force algorithms to be tuned without environment interaction.

Methods and Evaluation Protocols

Rather than proposing a new RL algorithm, this paper proposes a rigorous evaluation framework and benchmark suite.

The authors introduce a unified API to access diverse datasets and establish two distinct evaluation protocols to properly assess algorithm performance:

Online vs. Offline Policy Selection
  1. Online Policy Selection (Left): Algorithms are evaluated by interacting with the environment during the validation phase to find the best hyperparameters. This isolates the learning algorithm's potential but is unrealistic for real-world deployment.
  2. Offline Policy Selection (Right): Algorithms must be tuned without environment interaction. To benchmark this, the authors propose a naive baseline: they tune hyperparameters on a subset of "online" tasks, and then blindly apply those best hyperparameters to the reserved "offline" tasks within the same domain.

The authors evaluate several baseline algorithms using this framework, including standard Behavior Cloning (BC), standard off-policy methods (DQN, D4PG, IQN, REM), and specialized offline RL methods (BCQ, BRAC, RABM).

The RL Unplugged Datasets

The "RL Unplugged" suite includes datasets generated from a variety of domains, testing different axes of difficulty such as continuous versus discrete actions, partial observability, and exploration difficulty:

Task domains and their characteristics in RL Unplugged
  • DM Control Suite: Simulated continuous control tasks ranging from simple (Cartpole) to highly complex (Humanoid, Manipulator).
  • DM Locomotion: High-degree-of-freedom continuous control tasks featuring a CMU Humanoid and a virtual rodent. These tasks use egocentric vision (a camera attached to the agent), introducing partial observability and requiring memory.
  • Atari 2600: 46 discrete-action games. The datasets are generated by recording the replay buffer of an online DQN agent during training.
  • Real-world RL (RWRL) Suite: Tasks specifically designed to mimic real-world friction, including system delays, constraints, and non-stationarity.

Key Insights and Results

  • The Hyperparameter Trap: Many offline RL algorithms look fantastic when tuned using online simulator interactions. However, when subjected to strict offline policy selection, their performance often drops significantly, revealing a lack of robustness to hyperparameter choices.
  • Behavior Cloning is Surprisingly Strong: On the most difficult continuous control tasks (like making a complex humanoid run or a robotic arm insert a peg), simple Behavior Cloning (BC) is remarkably competitive. It often matches or outperforms complex, state-of-the-art offline RL algorithms.
  • Robustness Beats Specialization in Atari: Standard off-policy algorithms that are inherently robust (like REM and IQN) actually outperformed specialized offline RL algorithms (like BCQ) on Atari games when evaluated under the strict offline policy selection protocol. Standard DQN proved highly sensitive to hyperparameters, suffering massive performance drops.
  • Diversity Exposes Weaknesses: Evaluating algorithms only on simple tasks gives a false sense of progress. For example, D4PG works well on easy continuous tasks but fails completely on hard exploration or high-dimensional tasks. Furthermore, the authors noted they couldn't even get BCQ or BRAC to work well on the Locomotion tasks, highlighting the fragility of current offline methods.

Limitations and Future Work

The authors acknowledge that their proposed method for "offline policy selection" (transferring hyperparameters from similar tasks) is a naive baseline. They invite the community to develop better Offline Policy Evaluation (OPE) methods to solve this.

Across the board, no offline RL algorithm was able to reach the peak performance of online methods on the most difficult exploration tasks, showing there is still significant work to be done. For future work, the authors suggest that combining behavior regularization techniques with highly robust off-policy algorithms is a highly promising direction. Finally, they intend for RL Unplugged to be a "living" benchmark that grows with community dataset contributions.

Summary updated

Abstract

Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community. Moving forward, we view RL Unplugged as a living benchmark suite that will evolve and grow with datasets contributed by the research community and ourselves. Our project page is available on github.
BibTeX
@inproceedings{gulcehre2020rl,
  author = {Çaglar Gülçehre and Ziyu Wang and Alexander Novikov and Thomas Paine and Sergio Gómez Colmenarejo and Konrad Zolna and Rishabh Agarwal and Josh Merel and Daniel J. Mankowitz and Cosmin Paduraru and Gabriel Dulac-Arnold and Jerry Li and Mohammad Norouzi and Matthew Hoffman and Nicolas Heess and Nando de Freitas},
  title = {{RL Unplugged: A Collection of Benchmarks for Offline Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=vBBOYqEyNX-},
}
Figure