Fast Task Inference with Variational Intrinsic Successor Features

Steven Hansen, Will Dabney, André Barreto, David Warde-Farley, Tom Van de Wiele, Volodymyr Mnih

TL;DR

Variational Intrinsic Successor FeatuRes (VISR) is a reinforcement learning (RL) algorithm that bridges the gap between unsupervised skill discovery and fast task adaptation. By demonstrating that behavioral mutual information (BMI) can be used to learn the feature basis for Successor Features (SF), the authors enable agents to learn diverse behaviors without rewards. These agents can then "solve" new tasks in just minutes of interaction using simple linear regression.

The Challenge of Unsupervised Reinforcement Learning

Reinforcement learning is notoriously sample-inefficient, often requiring millions of interactions to learn a single task. While unsupervised pre-training has revolutionized computer vision and natural language processing, it has been significantly harder to apply to RL.

Existing "unsupervised RL" methods generally fall into two camps:

  1. Skill Discovery (e.g., DIAYN, VIC): These methods maximize the mutual information between a latent "skill code" and the states the agent visits. They learn diverse behaviors but struggle to adapt quickly when a specific reward function is finally introduced.
  2. Successor Features (SF): These allow for near-instant transfer between tasks, provided the reward function is a linear combination of some underlying features. However, these features usually have to be hand-engineered or learned from rewards, defeating the purpose of unsupervised pre-training.

The gap is clear: How can an agent automatically discover a set of features during an unsupervised phase that are perfectly suited for fast adaptation via Successor Features later?

Methods: The VISR Algorithm

The core contribution of this paper is the Variational Intrinsic Successor FeatuRes (VISR) algorithm. The authors demonstrate a mathematical synergy between two previously separate ideas.

First, they establish the BMI-SF Link: they show that the objective for maximizing Behavioral Mutual Information (learning distinguishable skills) can be adapted to learn the exact features ($\phi$) required by the Successor Features framework.

VISR Agent Architecture

To implement this, VISR uses a "Universal Successor Feature Approximator" (USFA). During a long unsupervised phase (e.g., 250 million frames), the agent explores the environment by sampling random "task vectors" ($w$) and trying to maximize an intrinsic reward defined by a discriminator. This discriminator ($q$) learns to map states to the features ($\phi$) that define those tasks.

When the agent is finally given an extrinsic reward (the "supervised phase"), it doesn't need to perform standard, slow RL. Instead, it collects a small amount of data, performs linear regression to find the weights ($w$) that map its learned features ($\phi$) to the observed rewards, and immediately computes an improved policy using Generalized Policy Improvement (GPI).

Key Insights

  • Duality of Discovery and Transfer: The most profound insight is that "learning to be different" (BMI) is functionally equivalent to "learning a feature basis for rewards" (SF). By framing skill discovery as feature learning, the agent prepares itself for any possible linear reward function.
  • Von Mises-Fisher Distribution: Unlike previous methods that used Gaussian distributions for latent skills, VISR uses the Von Mises-Fisher distribution. This constrains task vectors and features to a unit-sphere, which is a requirement for the Successor Feature math to remain stable.
  • Zero-shot to Few-shot: VISR effectively turns a reinforcement learning problem into a supervised learning problem (regression) at test time. This allows the agent to bypass the "trial and error" phase of standard RL for the downstream task.

Key Results

The authors evaluated VISR primarily on the Atari 57 Suite using a novel "two-phase" setup: 250 million steps of unsupervised pre-training followed by a tiny "test phase" of only 100,000 steps with rewards—an extremely low-data regime for Atari.

Performance comparison of VISR against baselines on Atari 57
  • Human-Level Performance: VISR achieved human-level performance on 12 Atari games using only 100k steps of reward-based training. For context, standard algorithms like DQN often require tens or hundreds of millions of steps to reach similar levels.
  • Outperforming Baselines: VISR significantly outperformed other unsupervised methods like DIAYN and curiosity-based models when adapted to the Atari suite.
  • Efficiency Gain: On several games (like James Bond or Krull), VISR's performance after 100k steps matched what a standard DQN agent would achieve only after 200 million frames of training.
  • Inference Superiority: The authors showed that using linear regression to infer the task vector was significantly more effective than "random search" (simply picking the best-performing skill from the unsupervised phase).

Limitations and Future Work

While VISR is excellent at learning to control the environment, its "pure" unsupervised exploration is slightly less aggressive than curiosity-based methods. The authors suggest that the agent might not visit the most distant or difficult-to-reach states without an explicit curiosity drive.

Additionally, VISR assumes that the downstream reward is a linear combination of its learned features. While this holds for many tasks, highly complex or sparse rewards might require non-linear adaptation. Future directions include combining VISR with curiosity-based intrinsic rewards to improve exploration during the pre-training phase, potentially allowing the agent to discover even more complex "controllable" features.

Summary updated

Abstract

We introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide fast task inference through the successor features framework.
BibTeX
@inproceedings{hansen2020fast,
  author = {Steven Hansen and Will Dabney and André Barreto and David Warde-Farley and Tom Van de Wiele and Volodymyr Mnih},
  title = {{Fast Task Inference with Variational Intrinsic Successor Features}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=SSg-lJ7G8b5},
}
Figure