Behavior From the Void: Unsupervised Active Pre-Training

Hao Liu, Pieter Abbeel

Introduction

Active Pre-Training (APT) is a novel framework for reinforcement learning (RL) that enables agents to learn useful behaviors and representations in an environment before ever encountering a reward signal. By maximizing a non-parametric entropy objective within a contrastively-learned latent space, APT allows agents to explore complex, high-dimensional environments—such as raw pixel inputs—far more effectively than prior methods. This unsupervised pre-training phase leads to massive gains in data efficiency for downstream tasks.

The Challenge of "Tabula Rasa" RL

Traditional RL typically starts from a "tabula rasa" state: the agent begins with zero knowledge and must stumble upon rewards to learn anything useful. This approach is incredibly data-inefficient compared to biological creatures, who spend much of their early lives exploring and "playing" without specific goals, building a repertoire of behaviors they can later adapt to solve new tasks quickly.

While unsupervised pre-training has revolutionized fields like Computer Vision and Natural Language Processing, it has historically struggled in RL. Previous attempts at "reward-free" exploration often relied on flawed paradigms:

  1. Pixel-level density modeling: Attempting to predict every pixel in an image is computationally expensive and often causes the agent to fixate on irrelevant, unpredictable noise (the "noisy TV" problem).
  2. Count-based exploration: Simply counting visited states works in simple grid worlds but fails to scale to the infinite variety of high-dimensional visual inputs.

The APT Methodology

The authors introduce APT to address these limitations by decoupling representation learning from behavior learning. The framework relies on two main components:

APT Architecture Diagram
  1. Contrastive Representation Learning: Instead of modeling pixels directly, APT uses a contrastive loss (similar to SimCLR) to map high-dimensional observations into a compact, abstract latent space. This ensures the agent focuses on meaningful, high-level features rather than pixel-level noise.
  2. Particle-Based Entropy Maximization: To encourage exploration, the agent receives an intrinsic reward based on the "novelty" of its current state. APT calculates this by measuring the distance between the current state's latent representation and its k-nearest neighbors in a replay buffer. A larger distance indicates a highly novel state, yielding a higher intrinsic reward.

By maximizing this "particle-based entropy," the agent is incentivized to push into the "void"—the unexplored regions of the state space—effectively covering the environment with a diverse set of learned behaviors.

Evaluation Datasets

The authors evaluated APT on two standard RL benchmarks:

  • DeepMind Control Suite (DMControl): A set of continuous control tasks (e.g., Cheetah, Hopper, Walker). The evaluation specifically included "Sparse Reward" versions of these tasks, which are notoriously difficult to solve from scratch.
  • Atari-57: A diverse suite of 57 classic video games, focusing on a 26-game subset for detailed comparison and the full 57-game set for benchmarking against state-of-the-art methods.

Key Insights

  • Latent Space is Key: Exploring directly in pixel space is a trap. By performing entropy maximization in a learned latent space, the agent ignores "distractor" noise and focuses on the underlying structure of the environment.
  • Non-parametric is Better: Unlike previous methods that tried to learn a complex, parametric probability density function to find novel states, APT's k-nearest neighbor approach is non-parametric. This makes it simpler, more stable, and highly scalable to high dimensions.
  • Task-Agnostic "Pre-training" Works: The paper demonstrates that an agent can spend millions of steps just "wandering" with an entropy objective and subsequently solve a specific task (like making a robot run) much faster than an agent that received task-specific rewards from step one.

Key Results

DMControl Performance Curves
  • DMControl: APT dramatically outperformed "from-scratch" RL (DrQ) and other exploration methods. On sparse reward tasks that are nearly impossible to solve normally, APT achieved high success rates with very few supervised samples.
  • Atari: In the data-limited setting (100k steps), APT achieved human-level performance on 12 games. It reached a median human-normalized score 3x higher than previous unsupervised RL methods (like VISR) and significantly improved upon the supervised state-of-the-art (DrQ).
  • Efficiency: On the Atari-26 subset, APT provided a 54% relative improvement in data efficiency over training from scratch.

Limitations and Future Work

  • Reward Scaling: APT works best when the scale of the intrinsic reward (exploration) matches the scale of the extrinsic reward (the task). If they are mismatched, the agent might ignore the task or stop exploring prematurely.
  • Computational Intensity: The pre-training phase is lengthy (e.g., 250M steps for Atari). While this is "reward-free" and can be done once for many downstream tasks, it still requires significant wall-clock time and compute resources.
  • Catastrophic Forgetting: When transitioning from the exploration phase to the task-specific phase, the agent can sometimes "forget" the diverse behaviors it learned. The authors suggest that better fine-tuning strategies or combining APT with model-based RL could mitigate this issue.

Summary updated

Abstract

We introduce a new unsupervised pre-training method for reinforcement learning called APT, which stands for Active Pre-Training. APT learns behaviors and representations by actively searching for novel states in reward-free environments. The key novel idea is to explore the environment by maximizing a non-parametric entropy computed in an abstract representation space, which avoids challenging density modeling and consequently allows our approach to scale much better in environments that have high-dimensional observations (e.g., image observations). We empirically evaluate APT by exposing task-specific reward after a long unsupervised pre-training phase. In Atari games, APT achieves human-level performance on 12 games and obtains highly competitive performance compared to canonical fully supervised RL algorithms. On DMControl suite, APT beats all baselines in terms of asymptotic performance and data efficiency and dramatically improves performance on tasks that are extremely difficult to train from scratch.
BibTeX
@inproceedings{liu2021behavior,
  author = {Hao Liu and Pieter Abbeel},
  title = {{Behavior From the Void: Unsupervised Active Pre-Training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=fIn4wLS2XzU},
}
Figure