In traditional reinforcement learning, an agent learns a value function that conflates two distinct types of information: the layout of the environment (where things are and how they move) and the current goal (what is rewarding). When the goal changes—even slightly—this monolithic value function collapses, forcing the agent to re-learn from scratch.
Successor Features (SF) solve this by factoring the value function into two independent components. This decomposition allows agents to adapt to new tasks nearly instantaneously by reusing their understanding of environmental dynamics.
The Value Decomposition
The core intuition, formalized by Andre Barreto, David Silver, and colleagues in Successor Features for Transfer in Reinforcement Learning, is to represent the value of a state-action pair $Q(s, a)$ as a dot product of two vectors:
- Successor Features ($\psi$): A predictive representation of the expected discounted future accumulation of state features. It answers: "If I follow this policy, what kind of features (e.g., colors, objects, locations) will I encounter in the future?"
- Reward Weights ($w$): A vector that defines the immediate reward associated with each feature. It answers: "How much do I currently care about each of those features?"
Mathematically, this is expressed as $Q(s, a) = \psi(s, a)^T w$. Because this relationship is linear, an agent can evaluate a previously learned policy on a brand-new task by simply updating the weight vector $w$ through linear regression, leaving the complex environmental dynamics captured in $\psi$ intact.
Why Decoupling Works
This separation bridges the gap between the speed of model-free RL and the flexibility of model-based planning. As Samuel J. Gershman and Tejas D. Kulkarni argue in Deep Successor Reinforcement Learning, SFs function as a "predictive map."
In a model-free system, if a reward moves from one room to another, the agent must propagate that change back through every state-action pair via Bellman updates. In an SF-based system, the agent already knows the "reachability" of the new room through its learned features. It only needs to realize that the reward weight for "Room B" is now high and "Room A" is now low. This allows for what Lucas Lehnert and Michael Littman describe as Model Reduction, where the agent clusters behaviorally equivalent states to transfer knowledge across tasks that differ in both rewards and transitions.
Generalized Policy Improvement
The true power of this decomposition emerges when combined with Generalized Policy Improvement (GPI). If an agent has a library of successor features for different policies, it can instantly compute the value of all those policies for a new task.
Andre Barreto demonstrated in Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement that an agent can act according to the best policy in its library for the current task, or even a combination of them. This often results in a policy that is better than any single policy in the library, even before the agent has taken a single new learning step.
Beyond Simple Rewards: Safety and Intent
The SF framework has proven robust enough to extend beyond simple navigation:
- Safety: In CAT: Caution Aware Transfer, Amy Zhang and Amrit Singh Bedi show that SFs can decouple risk structures from dynamics, allowing agents to adapt to novel hazards on the fly by re-weighting features associated with "caution."
- Inverse RL: Angelos Filos and Sergey Levine introduced PsiPhi-Learning, which uses SFs to learn from unlabeled demonstrations. By disentangling the demonstrator's goals from the environment's physics, the agent can "borrow" useful maneuvers even if the demonstrator was trying to achieve something entirely different.
- Unsupervised Pretraining: Methods like APS and the Forward-Backward (FB) representation by Ahmed Touati use SF-like measures to learn "reachability" without any rewards at all. This creates a "behavior foundation model" that can solve any user-specified reward function at inference time.
Intuition: The "Kitchen" Analogy
Imagine learning to navigate a kitchen. A standard RL agent learns that "moving left" is good because it leads to "coffee." If you suddenly want "tea" (located to the right), the agent is lost.
An agent using Successor Features learns the layout of the kitchen independently of its thirst. It learns that "moving left" leads to the "stove" and "moving right" leads to the "cupboard." When you tell the agent you want tea, it doesn't need to re-explore the kitchen. It simply updates its preference weights ($w$) for "cupboard" and "kettle," and its existing predictive map ($\psi$) immediately tells it which sequence of actions will maximize those features.