Introduction
Generalization remains a central challenge in Reinforcement Learning (RL). While agents can excel at specific tasks, transferring that knowledge to novel scenarios often requires significant retraining. Current approaches typically follow one of two paradigms: Parametric Generalization, such as Universal Value Function Approximators (UVFAs), which treat task descriptions as inputs to a neural network; and Structural Generalization, such as Successor Features (SFs) combined with Generalized Policy Improvement (GPI), which decouple environment dynamics from reward functions.
Universal Successor Features Approximators (USFAs) bridge this gap. By combining the scalability of UVFAs with the rapid policy evaluation of SFs, USFAs enable agents to generalize to entirely new tasks zero-shot. This is achieved by exploiting both the similarity between task goals and the underlying dynamics of the environment.
Methodology
The core innovation of USFAs is making Successor Features "universal." Standard SFs represent the expected future occupancy of features $\phi$ under a specific policy. USFAs extend this by conditioning the SFs on a policy encoding $z$.
The value of a state-action pair for a task $w$ using a policy $\pi_z$ is computed as:
$$Q(s, a, w, z) = \psi(s, a, z)^\top w$$
Where $\psi(s, a, z)$ are the Universal Successor Features learned via neural networks, and $w$ is the reward weight vector defining the task.
During training, the agent learns about various policies by sampling from a distribution $\mathcal{D}_z$. At test time, when presented with a novel task $w'$, the agent employs Generalized Policy Improvement (GPI). It evaluates a set of candidate policies $C$ instantly using the learned USFA and selects the best action across all of them. This allows the agent to "stitch together" behaviors from its training history to solve unseen tasks.
Experimental Evaluation
The authors evaluated USFAs using two primary setups: a simple Trip MDP for illustrative purposes and a complex 3D first-person environment in DeepMind Lab.
In the DeepMind Lab environment, the agent navigates a room containing four object types: TVs, balls, hats, and balloons. During training, the agent is exposed to "canonical" tasks, such as collecting a single object type. Testing involves "unseen" tasks with complex reward profiles, such as collecting two types while avoiding a third.
Key Insights and Results
USFAs demonstrate several advantages over traditional methods:
- Disentangling Task from Policy: By separating the reward vector ($w$) from the policy encoding ($z$), the agent can learn off-policy, gaining knowledge about behaviors it is not currently executing.
- Zero-Shot Adaptation: In DeepMind Lab, USFAs significantly outperformed standard UVFAs on unseen tasks. Notably, USFAs could generalize to tasks involving negative rewards (avoiding objects) even when only trained on positive rewards, as the structural understanding of object locations (the SFs) remains valid.
- Computational Efficiency: The architecture allows the agent to evaluate dozens of candidate policies in parallel with minimal extra cost, as policy conditioning occurs late in the network.
Discussion and Conclusion
While USFAs offer a powerful framework for transfer RL, they rely on the assumption that rewards are linear combinations of features ($\phi$). Furthermore, performance is sensitive to the quality of these features and the stability of off-policy learning. Future work may focus on the automated discovery of feature spaces directly from raw sensory input in more diverse, open-ended environments.
By providing a bridge between parametric and structural generalization, USFAs represent a significant step toward creating versatile RL agents capable of robust zero-shot transfer.
Summary updated