Abstract
It has been established that diverse behaviors spanning the controllable subspace of an Markov decision process can be trained by rewarding a policy for being distinguishable from other policies. However, one limitation of this formulation is generalizing behaviors beyond the finite set being explicitly learned, as is needed for use on subsequent tasks. Successor features provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor feature framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 14 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.
Introduction
Standard Reinforcement Learning (RL) is notoriously sample-inefficient, often requiring millions of interactions to learn tasks from scratch. Unsupervised pre-training offers a solution by allowing agents to learn environment dynamics and controllable behaviors before a specific reward is introduced. However, existing methods face two primary challenges: generalization beyond a finite set of learned skills and slow adaptation once a reward signal is finally provided.
Methodology: Variational Intrinsic Successor FeatuRes (VISR)
The authors introduce VISR, a framework that bridges Behavioral Mutual Information (BMI) and Successor Features (SF). BMI encourages the agent to learn a library of diverse, distinguishable behaviors by maximizing the mutual information between a latent goal vector and visited states. Successor Features decompose state values into state features ($\phi$) and a task vector ($w$), where learning a new task becomes a linear regression problem.
VISR's novelty lies in proving that the BMI objective can be adapted to learn the exact features required by the SF framework. By using a Von Mises-Fisher distribution for the discriminator, the model forces learned features to be unit-length and compatible with linear reward functions. During the adaptation phase, Generalized Policy Improvement (GPI) allows the agent to instantly combine multiple learned policies to solve new tasks more effectively than any single policy could alone.
Experimental Setup
The algorithm was evaluated on the Atari 57 suite using a novel setup: 250 million frames of unsupervised pre-training (no rewards), followed by a brief "test phase" of only 100,000 frames where rewards were visible. This regime tests the agent's ability to rapidly infer tasks and adapt behaviors from extremely limited feedback.
Results and Discussion
VISR demonstrates significant improvements in adaptation speed and final performance compared to existing unsupervised and supervised baselines.
As shown in the results table, the GPI VISR variant achieves human-level performance on 12 games in the full 57-game suite within just 100k steps of reward-based training. This significantly outperforms other unsupervised methods like DIAYN and even strong supervised baselines like Rainbow in this low-data regime. The use of linear regression to find the task vector $w$ proved far superior to simply searching through a library of pre-learned behaviors.
Key Insights
- Feature Discovery: BMI is treated as a method for discovering controllable features rather than just discrete skills. If an agent cannot influence a part of the state, it will not waste capacity representing it.
- Linearity for Speed: Reducing RL to a linear regression problem ($r = \phi^T w$) during adaptation enables near-instant task inference.
- Continuous Behavior Space: Unlike categorical skill methods, VISR learns a continuous space, allowing for interpolation between behaviors to solve novel tasks.
Conclusion
VISR represents a significant step toward agents that can learn effectively from limited feedback. While it excels at adaptation, the authors note that it is not necessarily better at discovering rare states than standard curiosity methods. Future work may integrate curiosity-based rewards to improve exploration during the unsupervised phase and ensure better coverage of the environment's state space.
Summary updated