Universal Successor Features Approximators

Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Hado van Hasselt, Rémi Munos, David Silver, Tom Schaul

Overview

Universal Successor Features Approximators (USFAs) combine the interpolation power of neural networks with the structural advantages of reinforcement learning to enable "zero-shot" transfer to new tasks. By learning a universal representation of how policies interact with environment dynamics, USFAs allow agents to instantly evaluate and combine previous skills to solve unseen problems without any additional training.

The Challenge of Zero-Shot Generalization

In Reinforcement Learning (RL), we often want an agent to master multiple tasks—like a robot that can both "pick up a cup" and "open a door." The holy grail is zero-shot generalization: the ability for an agent to perform a brand-new task the very first time it sees the reward function, without needing millions of trial-and-error steps.

Existing approaches usually fall into two camps:

  1. Universal Value Function Approximators (UVFAs): These treat the task description as just another input to a neural network. They are great at interpolating between tasks but treat the RL problem as a "black box," ignoring the underlying structure of rewards and dynamics.
  2. Successor Features (SFs) & Generalized Policy Improvement (GPI): This approach exploits the fact that rewards are often linear combinations of environmental features. SFs allow an agent to instantly calculate the value of an old policy under a new reward. GPI then lets the agent pick the best action among all its known policies. The limitation is that you are restricted to the specific library of policies you’ve already learned.

The authors of this paper ask: Can we combine the scalability of UVFAs with the structural efficiency of SFs?

Methods: Introducing USFAs

The paper introduces Universal Successor Features Approximators (USFAs). The core innovation is extending Successor Features to be "universal." Instead of learning a separate SF for every policy, the authors train a single function approximator $\tilde{\psi}(s, a, z)$ that predicts the successor features for any policy $\pi_z$.

USFA Architecture

Key components of the USFA framework include:

  • Policy Encoding: To make the SFs "universal," the model needs a way to represent a policy as a vector. It uses the reward weights $z$ that would make a policy optimal as its unique "ID."
  • Decoupled Training: Using an off-policy learning regime, the agent learns about many different policies $z$ simultaneously from the same stream of experience.
  • GPI at Inference: When faced with a new task $w$, the agent doesn't just use one policy. It uses the USFA to evaluate a set of candidate policies (the ones it trained on, plus others sampled near the new task) and follows the best one.

Experimental Setup

The researchers evaluated USFA on two distinct setups:

  1. Trip MDP: A simple, two-state "toy" environment designed to provide intuition. It simulates a traveler arriving in a new city who must choose between known coffee/food spots or exploring a neighborhood to find a better "mixed" option.
  2. DeepMind Lab: A large-scale 3D environment where an agent navigates a room containing four types of objects (TVs, balls, hats, and balloons). Tasks are defined by 4D reward vectors (e.g., $[1, -1, 0, 0]$ means "collect TVs, avoid balls"). This tests the agent's ability to handle high-dimensional visual inputs and complex navigation.

Key Insights and Results

USFA acts as a strict generalization of its predecessors. If you evaluate only the policy for the current task, you have a UVFA. If you evaluate a fixed set of training policies, you have SF&GPI. USFA allows you to operate anywhere on this spectrum.

  • Superior Zero-Shot Performance: In experiments, USFAs significantly outperformed standard UVFAs and traditional SF&GPI, closing the optimality gap much faster.
Optimality Gap Results
  • Disentangling Tasks and Policies: By separating the "task" (what we want to achieve) from the "policy" (how we behave), the agent can learn about potential behaviors it hasn't even fully executed yet.
  • Handling Avoidance: One of the most impressive findings was that USFAs could generalize to tasks involving negative rewards (avoiding certain objects) even if the training set only consisted of "positive" collection tasks.
  • The "Diversity" Boost: Including the training tasks in the candidate set for GPI is crucial. By having a diverse "library" of policies to choose from, the agent can almost always find a combination that works for a new task.

Limitations and Future Work

While powerful, the framework assumes that reward functions can be expressed as a linear combination of features ($\phi$). While these features can be learned, the performance of the system is heavily dependent on the quality of this representation. Furthermore, the choice of which policies $z$ to learn about during training and which to evaluate during testing is a major factor. Future work could explore more sophisticated "curriculum" approaches for policy selection, as well as methods to further stabilize off-policy training in highly dynamic environments.

Summary updated

Abstract

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.
BibTeX
@inproceedings{borsa2019universal,
  author = {Diana Borsa and André Barreto and John Quan and Daniel J. Mankowitz and Hado van Hasselt and Rémi Munos and David Silver and Tom Schaul},
  title = {{Universal Successor Features Approximators}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=N6cT0KRuNQ},
}
Figure