Combining Behaviors with the Successor Features Keyboard

Wilka Carvalho, Andre Saraiva, Angelos Filos, Andrew Kyle Lampinen, Loic Matthey, Richard L. Lewis, Honglak Lee, Satinder Singh, Danilo J. Rezende, Daniel Zoran

TL;DR

This paper introduces the Successor Features Keyboard (SFK), a framework that allows reinforcement learning agents to transfer behavioral knowledge to new, complex tasks without needing hand-designed features. By utilizing a novel Categorical Successor Feature Approximator (CSFA), the system jointly discovers state-features and task encodings, enabling the first successful demonstration of successor-feature-based transfer in a challenging 3D environment.

What Problem Does This Paper Address?

A major goal in AI is building agents that don't have to relearn everything from scratch. Successor Features (SFs) and Generalized Policy Improvement (GPI) provide a powerful mathematical framework for this: if an agent knows how to achieve certain "features" (like finding a red ball or moving to a door), it can instantly combine those behaviors to solve a new task (like "find the ball and bring it to the door").

However, previous implementations like the "Option Keyboard" had a major bottleneck: they required humans to hand-design the "cumulants" (the features the agent tracks) and the task encodings. This is labor-intensive and doesn't scale to complex, open-ended environments. While some work attempted to discover these representations, they often failed in high-dimensional 3D spaces or relied on static, non-dynamic transfer queries.

Methods

The authors propose two interconnected innovations:

  1. Successor Features Keyboard (SFK): This is the transfer mechanism. Unlike previous versions that used fixed queries, SFK learns a "transfer policy" that generates dynamic queries—linear combinations of known task behaviors—to adaptively select the best behavior at any given moment.
  2. Categorical Successor Feature Approximator (CSFA): This is the "discovery engine." Learning SFs while simultaneously discovering the features they are supposed to predict is notoriously unstable. CSFA solves this by modeling SFs as a probability mass function (PMF) over discretized bins (similar to the categorical approach in MuZero) rather than a single scalar point-estimate.
Figure 1: Diagram of transfer with the Successor Features Keyboard (SFK). SFK uses SF-based “keys” to represent behaviors by how much of some feature (known as a “cumulant”) they obtain. SFK dynamically selects from behaviors with GPI by generating prefer- ence “queries” over these features. The “value” of each behavior is then computed with a dot-product and the highest value is used to generate behavior. Prior work has hand-designed the features that define queries and SFs. Here, we study the problem of transferring with SFK while discovering all necessary representations.

To ensure stability during joint discovery, the authors also implemented two critical "tricks":

  • Bounding Task Encodings: Enforcing that task encodings lie on a unit sphere to prevent the magnitude of the vectors from blowing up the error bounds of the GPI.
  • Stop-Gradients: Preventing gradients from the Q-learning objective from flowing back into the task encoder, which otherwise leads to "dimensional collapse" where all tasks look the same to the agent.
Figure 2: Left: Categorical Successor Feature Approximator (CSFA) estimates SFs for a task encoding w with a structured, categorical network. Right: Successor Features Keyboard (SFK) transfers by dynamically selecting combinations of known task behaviors Wtrain = {w1, . . . , wn}. SFK accomplishes this by learning a policy gθ(s, wnew) that generates linear combinations of known task encodings Wtrain. SFK then leverages GPI to compute Q-values for known behaviors, Qwi,g = ψ(s, a, wi)⊤gθ(s, wnew) and acts using the highest Q-value.

Datasets

The researchers evaluated their methods in Playroom, a challenging 3D environment.

  • Observations: Partial, egocentric pixel-based images (what a robot would actually see).
  • Action Space: 46 discrete actions including rotation, movement, and object manipulation (picking up/holding).
  • Tasks: A curriculum of 32 training tasks, ranging from simple navigation ("Find A") to long-horizon manipulation ("Place A near B"). Transfer was tested on complex conjunctions of these tasks (e.g., "Put x 3," which requires completing three separate placement tasks).

Key Insights and Ideas

  • Categorical > Scalar: The most significant "aha moment" is that modeling Successor Features as distributions (PMFs) is essential for discovery. Because discovered cumulants are non-stationary (they change as the agent learns), scalar regression is too brittle. A PMF can represent the uncertainty and shifting magnitudes of these features much more robustly.
  • The Geometry of Transfer: The authors found that learned task encodings tend to cluster together in latent space. By sampling transfer queries as linear combinations (weighted sums) of known behaviors rather than arbitrary points in space, the agent stays within a "meaningful" region of its knowledge base.
  • Sharing is Caring: Sharing the SF-estimator and task encoder across all tasks—rather than having separate networks for each—is vital for scaling and representation re-use, but it requires the stability provided by CSFA to work.

Key Results

  • Discovery Success: CSFA was the only method capable of discovering representations that actually worked with GPI in the 3D Playroom. Baselines like USFA and MSFA could handle simple navigation but completely failed on long-horizon manipulation tasks.
  • Jumpstart Performance: In transfer experiments, SFK reached high performance significantly faster than standard Multitask RL (MTRL) or Distral. For the "Put x 3" task, SFK achieved similar performance to baselines while using 100 million fewer frames.
  • Zero-Curriculum Robustness: When the training curriculum was removed, the gap widened further; SFK was over 200 million frames faster than Distral, while MTRL failed to transfer entirely.
Figure 5: CSFA discovers representations compatible with GPI across short and long task-horizons. Both USFA and MSFA degrade in performance on Find tasks when paired with GPI. Neither are able to learn our longer horizon Place task. We hypothesize that this is because they approximate SFs with a point-estimate. (4 seeds)

Limitations and Future Work

  • Latent Space Concentration: The discovered task encodings were found to be very concentrated. The authors suggest that contrastive learning could be used in the future to encourage these representations to spread out, potentially making them more useful for a wider variety of tasks.
  • Temporal Abstraction: While SFK selects a new query at every single timestep, the authors note that employing "options" (behaviors that persist over time) and "initiation sets" could further improve performance and efficiency.
  • Scaling Limits: The study didn't test the upper limit of how many base tasks the encoder can handle, though the authors are optimistic about scaling to even more complex task combinations.

Summary updated

Abstract

The Option Keyboard (OK) was recently proposed as a method for transferring behavioral knowledge across tasks. OK transfers knowledge by adaptively combining subsets of known behaviors using Successor Features (SFs) and Generalized Policy Improvement (GPI). However, it relies on hand-designed state-features and task encodings which are cumbersome to design for every new environment. In this work, we propose the "Successor Features Keyboard" (SFK), which enables transfer with discovered state-features and task encodings. To enable discovery, we propose the "Categorical Successor Feature Approximator" (CSFA), a novel learning algorithm for estimating SFs while jointly discovering state-features and task encodings. With SFK and CSFA, we achieve the first demonstration of transfer with SFs in a challenging 3D environment where all the necessary representations are discovered. We first compare CSFA against other methods for approximating SFs and show that only CSFA discovers representations compatible with SF&GPI at this scale. We then compare SFK against transfer learning baselines and show that it transfers most quickly to long-horizon tasks.
BibTeX
@inproceedings{carvalho2023combining,
  author = {Wilka Carvalho and Andre Saraiva and Angelos Filos and Andrew Kyle Lampinen and Loic Matthey and Richard L. Lewis and Honglak Lee and Satinder Singh and Danilo J. Rezende and Daniel Zoran},
  title = {{Combining Behaviors with the Successor Features Keyboard}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=GhNCFtLSsy},
}
Figure