Successor Feature Decomposition for Transferable Reinforcement Learning

170 concepts105 papers1 proposal

In standard reinforcement learning, an agent learns a value function that conflates two distinct types of information: the physics of the environment (where actions lead) and the rewards of the task (what the agent wants). When the task changes, even if the environment's physics remain identical, this conflated representation typically collapses, forcing the agent to relearn from scratch.

Successor feature decomposition addresses this by explicitly decoupling environment dynamics from reward structures. By factorizing the value function into a task-agnostic representation of future state occupancy and a task-specific weight vector, agents can adapt to new objectives near-instantaneously.

The Anatomy of Decoupling

The core intuition, formalized by Barreto et al., is to assume that the immediate reward $r$ can be expressed as a linear combination of environment features $\phi$: $r(s, a, s') = \phi(s, a, s')^T w$. Here, $\phi$ represents "what happened" (e.g., the agent picked up an apple) and $w$ represents the agent's current preferences (e.g., how much it likes apples).

Under this assumption, the value function $Q(s, a)$—the expected discounted sum of future rewards—can be rewritten as: $$Q^\pi(s, a) = \psi^\pi(s, a)^T w$$ where $\psi^\pi(s, a) = \mathbb{E}^\pi [\sum_{t=0}^\infty \gamma^t \phi_{t+1} | s_0=s, a_0=a]$.

These Successor Features (SF), $\psi$, represent the expected future accumulation of features under a specific policy $\pi$. As Barreto et al. demonstrate, this decomposition allows for Generalized Policy Improvement (GPI). If an agent has a library of policies, it can compute the value of all of them for a brand-new task $w'$ simply by performing a dot product between their stored successor features and the new weights. The agent then follows the policy that yields the highest value at each state, often achieving near-optimal performance without any new environment interaction.

Learning Without Rewards

Because successor features capture the "where" of the environment rather than the "what is good," they are uniquely suited for unsupervised pretraining. Hansen et al. show that agents can learn a diverse basis of behaviors by maximizing the mutual information between states and latent tasks, effectively building a library of successor features before ever seeing a reward.

Once a downstream task is introduced, the agent only needs to solve a simple linear regression problem to find the weights $w$ that best explain the observed rewards. Liu et al. extended this with APS (Active Pretraining with Successor Features), combining entropy maximization with SFs to explore complex environments like Atari and adapt to new tasks with extreme data efficiency.

Compositional Intelligence and Safety

The linear structure of SFs enables agents to treat skills like "chords" on a keyboard. Barreto et al. introduced the Option Keyboard, which allows agents to synthesize novel behaviors by combining existing policies in the reward space. For instance, if an agent has a policy for "walking" and another for "grasping," it can instantly approximate a policy for "walking while grasping" by weighting the respective successor features.

This decoupling also provides a framework for handling constraints. Kim et al. developed Constrained GPI, which uses SFs to "sandwich" value estimates between rigorous bounds, preventing agents from making over-optimistic decisions in novel tasks. Similarly, Feng et al. showed that by representing both rewards and safety costs as successor features, agents can evaluate the safety of prior policies in new contexts using simple linear operations, facilitating zero-shot transfer that respects safety thresholds.

Scaling and Generalization

While early work focused on discrete actions, recent advancements have pushed SF decomposition into more complex domains:

  • Continuous Control: LI et al. extended SFs to continuous action spaces, allowing robots to transfer locomotion skills across tasks.
  • Multi-Agent Systems: Gupta et al. introduced UneVEn, which uses multi-agent successor features to solve coordination problems by learning a family of related tasks simultaneously.
  • Social Learning: Filos et al. proposed $\Psi\Phi$-learning, which allows agents to learn from the demonstrations of others by disentangling the demonstrator's goals from the environment's dynamics.

By separating the "physics" of the world from the "desires" of the agent, successor feature decomposition transforms reinforcement learning from a process of rigid habit formation into one of flexible, compositional planning. As Russek et al. argue, this mechanism may even bridge the gap between model-based planning and model-free execution in the brain, providing a computationally efficient way to cache the structure of the world.

Key Foundational Work

Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments (2016) — contributed 2 concept elements to this direction.
Deep Successor Reinforcement Learning (2016) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 121 papers
  2. Version 1 · 2026-10-04 · 143 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_successor_feature_decomposition_ee375afc,
  title = {successor feature decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/successor-feature-decomposition/rdr_ee375afc6ae24a399879daa7776f3484},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_successor_feature_decomposition_ee375afc_v2,
  title = {successor feature decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/successor-feature-decomposition/rdr_ee375afc6ae24a399879daa7776f3484/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake