Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning

Lucas Lehnert, Michael LittmanHttps://jmlr.csail.mit.edu/papers/v21/19-060.html 2025 · OpenReview

TL;DR

This paper introduces Linear Successor Feature Models (LSFMs) to bridge model-based and model-free reinforcement learning. LSFMs learn state representations that predict future reward outcomes, enabling generalization across tasks with varying dynamics and rewards, a key step towards more adaptable and efficient AI agents.

What Problem Does This Paper Address?

A fundamental challenge in reinforcement learning (RL) is enabling agents to generalize knowledge across different situations. When faced with high-dimensional sensory inputs (like a self-driving car's sensors), learning an optimal decision-making strategy for every possible input becomes computationally intractable – the "curse of dimensionality." To overcome this, agents compress this high-dimensional input into a lower-dimensional "latent state space." The crucial question is: what properties should these latent state representations have to facilitate effective generalization and learning, particularly when transferring knowledge between tasks? Existing methods often focus on either model-free (predicting values for a specific policy) or model-based (predicting future outcomes) approaches, but a unified understanding of how state representations connect these paradigms, especially for transfer learning, has been lacking.

Methods

The paper introduces Linear Successor Feature Models (LSFMs). LSFMs are a novel approach that connects successor features (SFs) to model-based RL. SFs, traditionally viewed as a model-free concept (predicting future state visitations under a policy), are here framed as a bridge to model-based learning. The core idea is that learning SFs is equivalent to learning representations that are predictive of future reward outcomes.

Key methodological contributions include:

  • Reward-Predictive State Representations: Defining and formalizing state representations that are predictive of future expected reward outcomes, independent of a specific policy.
  • Linear Successor Feature Models (LSFMs): A new model that learns state representations and SFs, enabling generalization across tasks with different transitions and rewards. This is achieved by learning a state representation that approximates SFs.
  • Connecting SFs to Model-Based RL: Demonstrating that LSFMs can be used to learn reward-predictive state representations, which are essential for model-based RL agents to predict future outcomes.
  • Analysis of Generalization: Investigating how different types of state representations (value-predictive vs. reward-predictive) generalize across tasks and policies.

Datasets

The paper uses several grid-world environments for experiments:

  • Column World Example: A 3x3 grid world used to illustrate state representation concepts.
  • Three-State MDP Example: Used to demonstrate empirical latent transition probabilities.
  • Puddle-World Task: A more complex grid-world task with probabilistic transitions and rewards, used to evaluate the learned representations and their ability to predict future rewards.
  • Transfer Grid Worlds (Task A and Task B): Two grid-world tasks designed to test the transferability of learned state representations across different reward functions and transition dynamics.
  • Combination Lock Tasks (Training, Test Lock 1, Test Lock 2): A set of tasks designed to evaluate how well reward-predictive state representations capture task-relevant information and generalize to variations in reward and transitions.

Key Insights and Ideas

  • SFs as a Bridge: Successor Features, often seen through a model-free lens, can be reframed to connect to model-based RL by learning representations predictive of future rewards.
  • Reward-Predictive Representations Generalize: State representations that focus on predicting future reward sequences (reward-predictive) are more adept at generalizing across tasks with different dynamics and reward structures than those focused solely on predicting a specific policy's value (value-predictive).
  • LSFMs Enable Model-Based RL: By learning reward-predictive state representations, LSFMs allow agents to build internal models of the task, crucial for model-based RL.
  • Value-Predictive Representations Can Overfit: Representations optimized for a specific policy's value function can "overfit" to that policy and task, hindering generalization to new tasks with different optimal policies.
  • Compression is Key for Generalization: Constraining the dimensionality of the latent state space forces the learning algorithm to find compressed representations that capture essential task structure, leading to better generalization.

Key Results

  • Improved Transfer Learning: LSFMs and reward-predictive state representations demonstrated significant improvements in transfer learning scenarios, allowing agents to learn optimal policies in new tasks with less data compared to value-predictive representations or tabular methods. For instance, in the transfer grid-world experiment, reusing a reward-predictive representation led to finding an optimal policy with significantly smaller data sets.
  • Accurate Reward Prediction: Experiments showed that learned reward-predictive state representations (using LSFMs and LAMs) produced significantly lower reward-prediction errors compared to random initializations.
  • Generalization Across Policies: The paper theoretically and empirically shows that reward-predictive state representations can generalize across the entire space of abstract policies, enabling agents to find optimal or near-optimal policies.
  • LSFMs vs. LAMs: While both LSFMs and Linear Action Models (LAMs) were shown to learn approximate reward-predictive state representations, LSFMs tended to produce lower prediction errors for expected reward outcomes.

Limitations and Future Work

  • Linearity Assumption: The current work focuses on linear state representations and models. Extending these ideas to non-linear latent transition and reward functions is left for future work.
  • Learning Neural Representations: The paper acknowledges that learning neural networks that map inputs to latent feature spaces predictive of future reward outcomes is beyond its scope and is an area for future research.
  • Data Efficiency and Exploration: While the paper demonstrates generalization, further investigation into efficient exploration algorithms and obtaining sample complexity or regret bounds similar to prior work is suggested.
  • Limited Training Data: The study notes that if the training data set is limited, it might not provide enough information to find an optimal policy, especially for value-predictive representations.
  • Bisimulation vs. Reward Prediction: The paper highlights that reward-predictive state representations do not necessarily encode bisimulation relations, which preserve more structure of the original MDP. This is a trade-off for generalization.

Summary updated

Abstract

A key question in reinforcement learning is how an intelligent agent can generalize knowledge across different inputs. By generalizing across different inputs, information learned for one input can be immediately reused for improving predictions for another input. Reusing information allows an agent to compute an optimal decision-making strategy using less data. State representation is a key element of the generalization process, compressing a high-dimensional input space into a low-dimensional latent state space. This article analyzes properties of different latent state spaces, leading to new connections between model-based and model-free reinforcement learning. Successor features, which predict frequencies of future observations, form a link between model-based and model-free learning: Learning to predict future expected reward outcomes, a key characteristic of model-based agents, is equivalent to learning successor features. Learning successor features is a form of temporal difference learning and is equivalent to learning to predict a single policy's utility, which is a characteristic of model-free agents. Drawing on the connection between model-based reinforcement learning and successor features, we demonstrate that representations that are predictive of future reward outcomes generalize across variations in both transitions and rewards. This result extends previous work on successor features, which is constrained to fixed transitions and assumes re-learning of the transferred state representation.
BibTeX
@inproceedings{lehnert2025successor,
  author = {Lucas Lehnert and Michael Littman},
  title = {{Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning}},
  booktitle = {Https://jmlr.csail.mit.edu/papers/v21/19-060.html},
  year = {2025},
  url = {https://openreview.net/forum?id=3SgBBJrc6F},
}
Figure