TL;DR
This paper introduces the Successor Features Keyboard (SFK), a framework that allows reinforcement learning agents to transfer behavioral knowledge to new, complex tasks without needing hand-designed features. By utilizing a novel Categorical Successor Feature Approximator (CSFA), the system jointly discovers state-features and task encodings, enabling the first successful demonstration of successor-feature-based transfer in a challenging 3D environment.
What Problem Does This Paper Address?
A major goal in AI is building agents that don't have to relearn everything from scratch. Successor Features (SFs) and Generalized Policy Improvement (GPI) provide a powerful mathematical framework for this: if an agent knows how to achieve certain "features" (like finding a red ball or moving to a door), it can instantly combine those behaviors to solve a new task (like "find the ball and bring it to the door").
However, previous implementations like the "Option Keyboard" had a major bottleneck: they required humans to hand-design the "cumulants" (the features the agent tracks) and the task encodings. This is labor-intensive and doesn't scale to complex, open-ended environments. While some work attempted to discover these representations, they often failed in high-dimensional 3D spaces or relied on static, non-dynamic transfer queries.
Methods
The authors propose two interconnected innovations:
- Successor Features Keyboard (SFK): This is the transfer mechanism. Unlike previous versions that used fixed queries, SFK learns a "transfer policy" that generates dynamic queries—linear combinations of known task behaviors—to adaptively select the best behavior at any given moment.
- Categorical Successor Feature Approximator (CSFA): This is the "discovery engine." Learning SFs while simultaneously discovering the features they are supposed to predict is notoriously unstable. CSFA solves this by modeling SFs as a probability mass function (PMF) over discretized bins (similar to the categorical approach in MuZero) rather than a single scalar point-estimate.
To ensure stability during joint discovery, the authors also implemented two critical "tricks":
- Bounding Task Encodings: Enforcing that task encodings lie on a unit sphere to prevent the magnitude of the vectors from blowing up the error bounds of the GPI.
- Stop-Gradients: Preventing gradients from the Q-learning objective from flowing back into the task encoder, which otherwise leads to "dimensional collapse" where all tasks look the same to the agent.
Datasets
The researchers evaluated their methods in Playroom, a challenging 3D environment.
- Observations: Partial, egocentric pixel-based images (what a robot would actually see).
- Action Space: 46 discrete actions including rotation, movement, and object manipulation (picking up/holding).
- Tasks: A curriculum of 32 training tasks, ranging from simple navigation ("Find A") to long-horizon manipulation ("Place A near B"). Transfer was tested on complex conjunctions of these tasks (e.g., "Put x 3," which requires completing three separate placement tasks).
Key Insights and Ideas
- Categorical > Scalar: The most significant "aha moment" is that modeling Successor Features as distributions (PMFs) is essential for discovery. Because discovered cumulants are non-stationary (they change as the agent learns), scalar regression is too brittle. A PMF can represent the uncertainty and shifting magnitudes of these features much more robustly.
- The Geometry of Transfer: The authors found that learned task encodings tend to cluster together in latent space. By sampling transfer queries as linear combinations (weighted sums) of known behaviors rather than arbitrary points in space, the agent stays within a "meaningful" region of its knowledge base.
- Sharing is Caring: Sharing the SF-estimator and task encoder across all tasks—rather than having separate networks for each—is vital for scaling and representation re-use, but it requires the stability provided by CSFA to work.
Key Results
- Discovery Success: CSFA was the only method capable of discovering representations that actually worked with GPI in the 3D Playroom. Baselines like USFA and MSFA could handle simple navigation but completely failed on long-horizon manipulation tasks.
- Jumpstart Performance: In transfer experiments, SFK reached high performance significantly faster than standard Multitask RL (MTRL) or Distral. For the "Put x 3" task, SFK achieved similar performance to baselines while using 100 million fewer frames.
- Zero-Curriculum Robustness: When the training curriculum was removed, the gap widened further; SFK was over 200 million frames faster than Distral, while MTRL failed to transfer entirely.
Limitations and Future Work
- Latent Space Concentration: The discovered task encodings were found to be very concentrated. The authors suggest that contrastive learning could be used in the future to encourage these representations to spread out, potentially making them more useful for a wider variety of tasks.
- Temporal Abstraction: While SFK selects a new query at every single timestep, the authors note that employing "options" (behaviors that persist over time) and "initiation sets" could further improve performance and efficiency.
- Scaling Limits: The study didn't test the upper limit of how many base tasks the encoder can handle, though the authors are optimistic about scaling to even more complex task combinations.
Summary updated