Successor Features (SFs) have emerged as a cornerstone of zero-shot reinforcement learning. By decoupling environment dynamics from specific reward functions, SFs allow an agent to adapt to new tasks instantly by simply projecting a new reward onto a pre-learned basis. However, the effectiveness of this approach hinges entirely on the choice of these "building block" features. While researchers have long relied on heuristics like Laplacian eigenfunctions, Yann Ollivier’s recent work provides the first rigorous mathematical derivation of the optimal features for Successor Features.
The Quest for Optimal Representations
In the standard SF framework, an agent explores an environment during a "pre-training" phase to learn a set of features. At "test time," when a new reward function is introduced, the agent can find a near-optimal policy without further exploration or planning.
The central challenge has always been: Which features should we learn?
Until now, the field has lacked a formal definition of "optimal." Most practitioners use Laplacian eigenfunctions because they capture the geometric connectivity of the environment and perform well empirically. Ollivier’s research moves beyond these heuristics, using the framework of entropy-regularized reinforcement learning to mathematically characterize the features that maximize performance across a broad spectrum of potential future tasks.
Methodology: A Task-Agnostic Approach
To find features that are truly general, the paper evaluates feature sets against three distinct, "prior-free" reward models:
- Gaussian White Noise: Dense, random rewards distributed throughout the state space.
- Goal-Reaching (Dirac): Sparse rewards located at a single, randomly chosen state.
- Scattered Rewards: A random mixture of sparse rewards with varying magnitudes and signs.
By analyzing the "Advantage Kernel"—a matrix representing how well a set of features can approximate the advantage function of any given task—the author identifies the eigenfunctions that maximize the expected return. This analysis is conducted in the context of entropy-regularized RL, where agents are penalized for deviating from a reference exploration policy, a setting increasingly common in offline RL and robotics foundation models.
Key Insights
1. The Universality of Features
Perhaps the most striking discovery is that the optimal features are identical across all three reward models. Whether the agent eventually faces dense noise or sparse goal-reaching tasks, the same underlying mathematical structures provide the best basis for Successor Features. This suggests that there is a fundamental "geometry of reinforcement learning" that is independent of the specific task type.
2. The Bellman Gap
A common practice in RL is to evaluate feature quality by minimizing the "Bellman error" (the consistency of the Q-function over a single time step). Ollivier proves that for Successor Features, the Bellman error is actually uninformative. The error depends primarily on the quantity of features used rather than their quality or suitability for solving downstream tasks. This finding explains why some representations with low Bellman error still fail to generalize well.
3. The Inverse Laplacian Connection
While Laplacian eigenfunctions are a good approximation, they are not strictly optimal. The paper demonstrates that in deterministic environments with long planning horizons (where the discount factor $\gamma$ approaches 1), the optimal features are eigenfunctions of a specific operator: $\Delta^{-1} + (\Delta^{-1})^*$.
This "inverse Laplacian" structure captures long-range dependencies more effectively than the standard Laplacian. This theoretical result provides a formal justification for the success of previous empirical methods, such as Forward-Backward representations, which implicitly approximate these inverse structures.
4. Horizon-Dependent Frequency
The optimal features are sensitive to the planning horizon ($\gamma$). When $\gamma$ is large, the agent requires "low-frequency" features that capture large-scale environmental structures. Conversely, when $\gamma$ is small (short-term planning), the agent needs "high-frequency" features to distinguish between immediate local actions.
Mathematical Results
The paper provides explicit formulas for these optimal features. In the general case, they are the largest eigenfunctions of a matrix derived from transition probabilities and the environment's stationary distribution. For deterministic environments, the optimal features are the largest eigenfunctions of:
$$\Delta^{-1} + (\Delta^{-1})^* - (1-\gamma^2)(\Delta^{-1})^*\Delta^{-1}$$
where $\Delta$ is the Laplacian operator.
Limitations and Future Directions
While this work provides a rigorous theoretical foundation, it operates under the assumption of "high regularization" (the large temperature limit), where the agent's policy remains close to the exploration policy. Future research is needed to determine if these features remain optimal when the agent is allowed to deviate significantly from the reference policy.
Additionally, the paper characterizes the ideal features but does not yet provide a scalable deep learning algorithm to extract them from high-dimensional inputs like raw pixels. Finally, because Successor Features are inherently linear, they may still struggle with tasks that are highly non-linear relative to the feature space, suggesting a need for future exploration into non-linear task encodings.
Summary updated