Policy Representations via Long-Term Behavioral Footprints

25 concepts14 papers2 proposals

In reinforcement learning (RL), we frequently need to compare different policies. Whether we are trying to select the best policy from a pool of offline candidates, discover a diverse set of skills, or recognize the goals of an observed agent, we need a answer to a fundamental question: How similar are two behaviors?

The naive approach is to compare policy parameters or neural network weights, but deep networks can look entirely different structurally while producing identical behavior. Another common alternative is to measure the divergence between immediate action distributions (e.g., using KL-divergence) across a set of states. However, action-space metrics are myopic. Two driving policies might choose different steering angles at a specific fork in the road but ultimately arrive at the same destination via equally safe routes. Conversely, taking the exact same action in a chaotic environment might lead to wildly different outcomes later.

To truly compare, select, or diversify policies, we need a way to map a policy to a compact representation that captures its long-term environmental footprint.

Successor Features as Behavioral Signatures

The core mechanism enabling this shift from action-space to behavior-space is the concept of Successor Features (SFs).

In standard RL, value functions predict the discounted sum of future rewards. Successor features generalize this concept: instead of predicting a scalar reward, they predict the discounted future occupancy of a vector of state features $\phi(s)$. Mathematically, the successor features $\psi^\pi(s,a)$ for a policy $\pi$ are defined as:

$$\psi^\pi(s,a) = \mathbb{E}_\pi \left[ \sum_{t=0}^\infty \gamma^t \phi(s_t) \Big| s_0=s, a_0=a \right]$$

This vector acts as a statistical summary of a trajectory. It does not just record what an agent does in the current state; it encodes where the policy will systematically guide the agent over an extended horizon.

As explored by Scarpellini et al. in their $\pi$2vec framework, representing policies by how they alter the environment over time provides a robust, comparable feature vector. This behavioral footprint distinguishes meaningful, goal-directed policies from random exploration and avoids the pitfalls of action-based comparisons that fail when different action sequences yield identical environmental outcomes.

Why Behavioral Metrics Work

By embedding policies into an SF space, we can compute geometric distances (such as the $L_2$ distance) between different policies. If the distance $\|\psi^{\pi_1} - \psi^{\pi_2}\|$ is small, it guarantees that under both policies, the agent expects to experience the environment's features in a nearly identical way over the long run.

This geometric property makes downstream optimization highly efficient. For instance, consider the problem of Offline Policy Selection (OPS), where an practitioner must identify the best-performing policy from a large candidate pool using only historical data. Testing every policy in the real world is prohibitively expensive.

To solve this, Konyushkova et al. utilize a policy-similarity kernel to propagate performance information across a candidate pool. If two policies exhibit similar behavioral patterns on a shared dataset, their real-world performance is highly likely to be correlated. By evaluating a single policy online, a surrogate model can infer the performance of behaviorally similar policies. Scarpellini et al. scale this intuition by combining successor features with visual foundation models, allowing accurate performance prediction entirely offline without live robot testing.

[Policy Space] ──(Weights/Actions)──> Myopic, high variance
[SF Mapping]   ──(Expected Occupancy)─> Compact Behavioral Vector
[Distance Metric] ────────────────────> Clear, geometric similarity

Discovering and Pruning for Diversity

The ability to quantify behavioral distance is equally transformative for Quality-Diversity (QD) algorithms and unsupervised skill discovery. Standard RL algorithms typically converge to a single optimal policy, but complex tasks often have multiple distinct, high-performing solutions (e.g., different gaits for a walking robot).

Zahavy et al. formalized this search through Diverse Successive Policies (DSP). By treating the search as a Constrained Markov Decision Process, the authors use successor features to capture the stationary distribution of a policy, ensuring that discovered behaviors are qualitatively different yet nearly optimal. Similarly, Grillotti et al. developed the Quality-Diversity Actor-Critic (QDAC) framework, which deploys two distinct critics: one for task performance and one for successor features. This allows the agent to optimize directly for long-term behavioral signatures rather than relying on instantaneous state-based differences.

This framework also prevents agents from hoarding redundant behaviors. In unsupervised skill discovery, agents frequently learn skills that look different in action space but achieve identical results. Lee et al. use successor features to evaluate skills based on their long-term behavioral impact, allowing a master policy to prune redundant skills based on functional intent. In purely offline settings, Kolev et al. demonstrate that calculating distances via successor features allows for stable diversity maximization directly from static datasets, bypassing the need for unstable adversarial skill discriminators.

Alternative Granularities of Behavioral Similarity

While successor features excel at capturing long-horizon environmental impacts, the broader paradigm of behavioral metrics spans multiple granularities depending on the problem context:

  • Intent and Goal Recognition: If we want to understand what an agent is trying to achieve, we can measure how well its observed actions align with a theoretical policy optimized for a specific target. Amado et al. utilize distance measures like KL-divergence to rank candidate goals based on observed actions. Taking this further, Cano et al. show that intent itself can be quantified as a measurable degree of optimization toward a goal by calculating the proximity between an agent's actual policy and an idealized optimal policy.
  • Trajectory Sequence Matching: When modeling complex biological decision-making where strategies shift over time, sequence-level metrics are vital. Claeys et al. employ trajectory similarity metrics like Dynamic Time Warping (DTW) and Wasserstein distances to map observed behavioral sequences to candidate strategies, providing an interpretable window into decision-making.
  • Transition-Level Corrections: For some algorithmic challenges, long-term dependencies are a hindrance rather than a help. To mitigate off-policy bias in actor-critic methods, Saglam et al. introduce a parameter-free similarity metric that operates on individual transitions. This quantifies the divergence between behavioral and target policies at each step, reducing computational overhead while stabilizing learning.
  • Task and Representation Learning: Behavioral metrics can also guide how an agent perceives its environment. Gutiérrez et al. measure the divergence of optimal policies across different tasks to identify and remove redundant training tasks in meta-RL. To learn better state representations, Liao et al. introduce a policy-independent metric that looks at the worst-case reward sequences achievable from state-action pairs. Finally, for Off-Policy Evaluation (OPE), Pavse et al. demonstrate that general control metrics fall short; instead, tailoring a similarity metric specifically to the evaluation policy's expected future returns yields representations that capture the underlying action-value function with significantly higher fidelity.

Key Foundational Work

Active Offline Policy Selection (2021) — contributed 2 concept elements to this direction.
Information-theoretic Task Selection for Meta-Reinforcement Learning (2020) — contributed 1 concept elements to this direction.
Discovering Diverse Nearly Optimal Policies withSuccessor Features (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_successor_feature_representation_ce4952ca,
  title = {successor feature representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/successor-feature-representation/rdr_ce4952ca370a486c85934804919f2638},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_successor_feature_representation_ce4952ca_v1,
  title = {successor feature representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/successor-feature-representation/rdr_ce4952ca370a486c85934804919f2638/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake