In standard reinforcement learning (RL), an agent learns a policy to maximize a specific reward function. If the goal changes—even slightly—the agent typically must start learning from scratch. This lack of flexibility is a major bottleneck for building general-purpose robots or assistants.
A growing body of research suggests a more efficient path: Compositional Reinforcement Learning. Instead of learning monolithic policies, agents can learn a library of "basis" behaviors and instantly combine them to solve new, unseen tasks. This zero-shot transfer is made possible by two core mechanisms: Successor Features and Generalized Policy Improvement.
Decoupling Dynamics from Rewards
To reuse knowledge, an agent must separate "how the world works" (dynamics) from "what it is trying to achieve" (rewards). Barreto et al. formalized this using Successor Features (SF).
In the SF framework, the reward for a task is represented as a linear combination of environment features $\phi$. If we know the weights $w$ for a new task, the value of a policy is simply the dot product of those weights and the agent’s "expected future features"—the Successor Features. As Carvalho et al. argue, these predictive representations act as the fundamental building blocks of intelligence, allowing an agent to cache expectations about the future rather than just immediate scalar rewards.
The Power of the Max Operator
Once an agent has a library of policies (and their corresponding Successor Features), how does it use them for a new task? The answer is Generalized Policy Improvement (GPI).
Barreto et al. proved that if you have a set of policies, you can define a new policy that, at every state, chooses the action recommended by the policy with the highest expected value for the current task. Crucially, this synthesized policy is guaranteed to perform at least as well as the best individual policy in the library.
This mechanism allows for instantaneous adaptation. For example, Zhang et al. provided theoretical guarantees that this approach converges faster than learning from scratch, while Alver et al. demonstrated that a small set of "independent policies" can act as a behavior basis to solve any task that is a linear combination of the underlying features.
Linear Task Composition
The most striking property of this research direction is that task composition often becomes a simple linear algebra problem.
In specific frameworks like Linearly Solvable MDPs (LMDPs) or entropy-regularized RL, the Bellman equations become linear. Saxe et al. and van Niekerk et al. showed that in these settings, the optimal value function for a composite task (like "find the key OR find the door") is a linear combination of the value functions of the individual tasks.
This allows for:
- Zero-shot synthesis: Combining pre-learned primitives to solve complex spatial tasks (Saxe et al.).
- Preference adaptation: Instantly adjusting to new trade-offs in multi-objective environments without retraining (Bazzan et al.).
- Language grounding: Mapping natural language instructions to linear weights that activate the relevant basis behaviors (Sikchi et al.).
Building a Better Basis
The efficiency of this approach depends on the quality of the policy library. If the library is sparse, the agent might struggle. Recent work has focused on how to construct the "optimal" basis:
- Unsupervised Discovery: Agarwal et al. and Bagot et al. propose learning basis functions from reward-free data by clustering the state space or learning geometric "behavior spaces."
- Modular Approximators: Carvalho et al. introduced Modular Successor Feature Approximators (MSFA), which discover important environment features automatically, allowing agents to avoid new obstacles while pursuing multiple items in a zero-shot manner.
- Hybrid Planning: To handle gaps in the library, Alegre et al. developed $h$-GPI, which uses a short-horizon environment model to look ahead before bootstrapping from the existing policy library, making the system robust to incomplete knowledge.
By treating reinforcement learning as the composition of predictive building blocks rather than the optimization of a single goal, this research direction provides a mathematically grounded path toward agents that can learn once and act anywhere.
Key Foundational Work
Mixture Model POMDPs for Efficient Handling of Uncertainty in Dialogue Management (2008) — contributed 1 concept elements to this direction.
Bayesian Policy Reuse (2015) — contributed 1 concept elements to this direction.