TL;DR
APS (Active Pretraining with Successor Features) is a novel unsupervised reinforcement learning (RL) framework that bridges the gap between exploration and exploitation. By combining the exploratory power of entropy maximization with the rapid task-adaptation of successor features, APS allows agents to explore complex environments without rewards and subsequently solve downstream tasks with extreme data efficiency. It significantly outperforms prior state-of-the-art methods on the Atari benchmark.
The Problem: The "Tabula Rasa" Paradigm in RL
In fields like Natural Language Processing and Computer Vision, "pretraining" is the gold standard: models learn general features from massive unlabeled datasets and are then fine-tuned for specific tasks. Reinforcement Learning, however, still largely relies on the "tabula rasa" paradigm—training an agent from scratch for every new goal.
Unsupervised RL aims to change this by allowing an agent to interact with an environment without a reward signal to "learn how the world works." Current approaches generally fall into two camps, both with distinct flaws:
- Entropy Maximization (e.g., APT): These agents try to visit as many diverse states as possible. They are great explorers but struggle to turn that experience into specific, exploitable skills.
- Mutual Information Maximization (e.g., VISR): These agents try to learn distinct behaviors (skills) associated with different "task vectors." They are great at adapting to new rewards but often fail to explore enough of the environment to find those rewards in the first place.
This paper addresses this "exploration-exploitation" gap in unsupervised pretraining by proposing a method that simultaneously achieves both.
Methods: Bridging Exploration and Skill Discovery
The core idea of APS is to maximize the Mutual Information $I(s; z)$ between states ($s$) and latent task variables ($z$). Using a standard information-theoretic identity, the authors decompose this objective into two distinct parts:
$$I(s; z) = H(s) - H(s|z)$$
- The "Active" Part ($H(s)$): This is the marginal entropy of states. Maximizing this encourages the agent to be "active"—to go out and find new, unseen states. APS handles this using a non-parametric, particle-based entropy estimator (borrowed from the APT algorithm).
- The "Successor" Part ($-H(s|z)$): This is the conditional entropy. Minimizing this ensures that for a given task $z$, the agent's behavior is predictable and distinct. APS optimizes this by using Successor Features (SFs). SFs are a clever way to represent the environment's dynamics such that the value of any task can be computed as a simple linear combination of features.
The "Best of Both Worlds" Intrinsic Reward: The agent receives a reward that is the sum of an exploration bonus (how unique is this state?) and a task-inference bonus (how well does this state match the current task vector?). This allows the agent to explore the environment while simultaneously organizing that experience into a library of reusable skills.
Key Insights
- Mutual Information Decomposition: The most elegant insight is that the mathematical decomposition $I(s; z) = H(s) - H(s|z)$ perfectly maps to the two biggest needs in unsupervised RL: exploration ($H(s)$) and skill discovery ($H(s|z)$).
- Successor Features as Discriminators: Usually, mutual information methods use a "classifier" to guess the task $z$ from the state $s$. APS demonstrates that Successor Features can act as a more powerful, RL-native version of this discriminator.
- Mutual Reinforcement: The authors found that exploration and skill learning aren't just additive; they are synergistic. Better exploration provides more diverse data to train the successor features, and better successor features help the agent navigate more effectively to distant, high-entropy regions.
Datasets and Key Results
The authors evaluated APS across custom GridWorlds designed to test exploration bottlenecks, the Atari 100k Benchmark (testing extreme data efficiency with only ~2 hours of gameplay), and the full Atari 57 Suite.
- Atari 100k: On the 26-game benchmark, APS achieved a median human-normalized score of 58.8, significantly outperforming the previous state-of-the-art, APT (47.5), and the skill-discovery method VISR (6.59).
- Data Efficiency: On the full Atari 57 suite, APS achieved a median score 3x higher than a standard DQN trained with 100x more supervised environment steps (10M vs 100k).
- Solving the "Hard" GridWorld: In custom environments, APS was the only method that could both find the keys (exploration) and use them to reach the goal (task inference).
Limitations and Future Work
While APS represents a significant leap forward for unsupervised RL, several avenues for improvement remain:
- Representation Learning: APS currently relies on a shared encoder and contrastive learning. Integrating more advanced representation learning, such as inverse dynamic models, could further boost performance.
- Task Vector Sampling: The task vectors $z$ are currently sampled from a uniform distribution on a sphere. Future work could investigate "curriculum learning" for these tasks—focusing on tasks that are at the edge of the agent's current abilities.
- Continuous Control: While tested extensively on discrete Atari games, the framework is theoretically applicable to continuous robotic control, which remains an open and exciting area for future testing.
Summary updated