Universal Successor Features Approximators

Diana Borsa, André Barreto, John Quan, Daniel J. Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, Tom Schaul

Introduction

Generalization remains a central challenge in Reinforcement Learning (RL). While agents can excel at specific tasks, transferring that knowledge to novel scenarios often requires significant retraining. Current approaches typically follow one of two paradigms: Parametric Generalization, such as Universal Value Function Approximators (UVFAs), which treat task descriptions as inputs to a neural network; and Structural Generalization, such as Successor Features (SFs) combined with Generalized Policy Improvement (GPI), which decouple environment dynamics from reward functions.

Universal Successor Features Approximators (USFAs) bridge this gap. By combining the scalability of UVFAs with the rapid policy evaluation of SFs, USFAs enable agents to generalize to entirely new tasks zero-shot. This is achieved by exploiting both the similarity between task goals and the underlying dynamics of the environment.

Methodology

The core innovation of USFAs is making Successor Features "universal." Standard SFs represent the expected future occupancy of features $\phi$ under a specific policy. USFAs extend this by conditioning the SFs on a policy encoding $z$.

The value of a state-action pair for a task $w$ using a policy $\pi_z$ is computed as:

$$Q(s, a, w, z) = \psi(s, a, z)^\top w$$

Where $\psi(s, a, z)$ are the Universal Successor Features learned via neural networks, and $w$ is the reward weight vector defining the task.

USFA architecture and learning algorithm.

During training, the agent learns about various policies by sampling from a distribution $\mathcal{D}_z$. At test time, when presented with a novel task $w'$, the agent employs Generalized Policy Improvement (GPI). It evaluates a set of candidate policies $C$ instantly using the learned USFA and selects the best action across all of them. This allows the agent to "stitch together" behaviors from its training history to solve unseen tasks.

Experimental Evaluation

The authors evaluated USFAs using two primary setups: a simple Trip MDP for illustrative purposes and a complex 3D first-person environment in DeepMind Lab.

In the DeepMind Lab environment, the agent navigates a room containing four object types: TVs, balls, hats, and balloons. During training, the agent is exposed to "canonical" tasks, such as collecting a single object type. Testing involves "unseen" tasks with complex reward profiles, such as collecting two types while avoiding a third.

Training tasks and object features in the DeepMind Lab environment.

Key Insights and Results

USFAs demonstrate several advantages over traditional methods:

  • Disentangling Task from Policy: By separating the reward vector ($w$) from the policy encoding ($z$), the agent can learn off-policy, gaining knowledge about behaviors it is not currently executing.
  • Zero-Shot Adaptation: In DeepMind Lab, USFAs significantly outperformed standard UVFAs on unseen tasks. Notably, USFAs could generalize to tasks involving negative rewards (avoiding objects) even when only trained on positive rewards, as the structural understanding of object locations (the SFs) remains valid.
  • Computational Efficiency: The architecture allows the agent to evaluate dozens of candidate policies in parallel with minimal extra cost, as policy conditioning occurs late in the network.

Discussion and Conclusion

While USFAs offer a powerful framework for transfer RL, they rely on the assumption that rewards are linear combinations of features ($\phi$). Furthermore, performance is sensitive to the quality of these features and the stability of off-policy learning. Future work may focus on the automated discovery of feature spaces directly from raw sensory input in more diverse, open-ended environments.

By providing a bridge between parametric and structural generalization, USFAs represent a significant step toward creating versatile RL agents capable of robust zero-shot transfer.

Summary updated

Abstract

The ability of a reinforcement learning (RL) agent to learn about many reward functions at the same time has many potential benefits, such as the decomposition of complex tasks into simpler ones, the exchange of information between tasks, and the reuse of skills. We focus on one aspect in particular, namely the ability to generalise to unseen tasks. Parametric generalisation relies on the interpolation power of a function approximator that is given the task description as input; one of its most common form are universal value function approximators (UVFAs). Another way to generalise to new tasks is to exploit structure in the RL problem itself. Generalised policy improvement (GPI) combines solutions of previous tasks into a policy for the unseen task; this relies on instantaneous policy evaluation of old policies under the new reward function, which is made possible through successor features (SFs). Our proposed universal successor features approximators (USFAs) combine the advantages of all of these, namely the scalability of UVFAs, the instant inference of SFs, and the strong generalisation of GPI. We discuss the challenges involved in training a USFA, its generalisation properties and demonstrate its practical benefits and transfer abilities on a large-scale domain in which the agent has to navigate in a first-person perspective three-dimensional environment.
BibTeX
@inproceedings{borsa2019universal,
  author = {Diana Borsa and André Barreto and John Quan and Daniel J. Mankowitz and Rémi Munos and Hado van Hasselt and David Silver and Tom Schaul},
  title = {{Universal Successor Features Approximators}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=N6cT0KRuNQ},
}
Figure