Overview
"RL Unplugged" introduces a comprehensive suite of benchmarks and datasets for offline reinforcement learning (RL) to address the field's reproducibility crisis. By standardizing diverse environments—from Atari games to complex robotic locomotion—and establishing strict evaluation protocols, the authors provide a robust foundation for developing RL algorithms that learn from fixed datasets without requiring costly or dangerous real-world exploration.
The Challenge of Real-World RL
While Reinforcement Learning has achieved massive success in simulated environments (like Go or StarCraft), its adoption in the real world remains limited. Real-world systems like power plants, healthcare systems, or autonomous vehicles cannot safely or cheaply support the millions of trial-and-error interactions required by standard "online" RL.
Offline RL solves this by training agents entirely on previously logged datasets. However, offline RL research is currently plagued by a reproducibility crisis. Algorithms are often evaluated on non-standardized datasets, and more problematically, researchers frequently tune their offline algorithms using online interactions with a simulator (to see which hyperparameters work best). This defeats the purpose of offline RL, giving an overly optimistic view of how these algorithms would perform in the real world where such tuning is impossible. This paper addresses this gap by providing standardized datasets, unified APIs, and strict evaluation protocols that force algorithms to be tuned without environment interaction.
Methods and Evaluation Protocols
Rather than proposing a new RL algorithm, this paper proposes a rigorous evaluation framework and benchmark suite.
The authors introduce a unified API to access diverse datasets and establish two distinct evaluation protocols to properly assess algorithm performance:
- Online Policy Selection (Left): Algorithms are evaluated by interacting with the environment during the validation phase to find the best hyperparameters. This isolates the learning algorithm's potential but is unrealistic for real-world deployment.
- Offline Policy Selection (Right): Algorithms must be tuned without environment interaction. To benchmark this, the authors propose a naive baseline: they tune hyperparameters on a subset of "online" tasks, and then blindly apply those best hyperparameters to the reserved "offline" tasks within the same domain.
The authors evaluate several baseline algorithms using this framework, including standard Behavior Cloning (BC), standard off-policy methods (DQN, D4PG, IQN, REM), and specialized offline RL methods (BCQ, BRAC, RABM).
The RL Unplugged Datasets
The "RL Unplugged" suite includes datasets generated from a variety of domains, testing different axes of difficulty such as continuous versus discrete actions, partial observability, and exploration difficulty:
- DM Control Suite: Simulated continuous control tasks ranging from simple (Cartpole) to highly complex (Humanoid, Manipulator).
- DM Locomotion: High-degree-of-freedom continuous control tasks featuring a CMU Humanoid and a virtual rodent. These tasks use egocentric vision (a camera attached to the agent), introducing partial observability and requiring memory.
- Atari 2600: 46 discrete-action games. The datasets are generated by recording the replay buffer of an online DQN agent during training.
- Real-world RL (RWRL) Suite: Tasks specifically designed to mimic real-world friction, including system delays, constraints, and non-stationarity.
Key Insights and Results
- The Hyperparameter Trap: Many offline RL algorithms look fantastic when tuned using online simulator interactions. However, when subjected to strict offline policy selection, their performance often drops significantly, revealing a lack of robustness to hyperparameter choices.
- Behavior Cloning is Surprisingly Strong: On the most difficult continuous control tasks (like making a complex humanoid run or a robotic arm insert a peg), simple Behavior Cloning (BC) is remarkably competitive. It often matches or outperforms complex, state-of-the-art offline RL algorithms.
- Robustness Beats Specialization in Atari: Standard off-policy algorithms that are inherently robust (like REM and IQN) actually outperformed specialized offline RL algorithms (like BCQ) on Atari games when evaluated under the strict offline policy selection protocol. Standard DQN proved highly sensitive to hyperparameters, suffering massive performance drops.
- Diversity Exposes Weaknesses: Evaluating algorithms only on simple tasks gives a false sense of progress. For example, D4PG works well on easy continuous tasks but fails completely on hard exploration or high-dimensional tasks. Furthermore, the authors noted they couldn't even get BCQ or BRAC to work well on the Locomotion tasks, highlighting the fragility of current offline methods.
Limitations and Future Work
The authors acknowledge that their proposed method for "offline policy selection" (transferring hyperparameters from similar tasks) is a naive baseline. They invite the community to develop better Offline Policy Evaluation (OPE) methods to solve this.
Across the board, no offline RL algorithm was able to reach the peak performance of online methods on the most difficult exploration tasks, showing there is still significant work to be done. For future work, the authors suggest that combining behavior regularization techniques with highly robust off-policy algorithms is a highly promising direction. Finally, they intend for RL Unplugged to be a "living" benchmark that grows with community dataset contributions.
Summary updated