Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning

Lucas Lehnert, Michael L. LittmanJMLR 2020

TL;DR

This paper introduces Linear Successor Feature Models (LSFMs) that bridge the gap between model-free and model-based reinforcement learning. LSFMs learn state representations that predict future reward outcomes, enabling agents to generalize knowledge across tasks with varying dynamics and rewards, a significant step towards more adaptable and data-efficient AI.

What Problem Does This Paper Address?

A fundamental challenge in reinforcement learning (RL) is how intelligent agents can generalize knowledge learned in one situation to new, unseen scenarios. This is crucial for efficiency, as it allows agents to learn optimal strategies with less data. High-dimensional sensory inputs can exacerbate this problem, leading to the "curse of dimensionality." While existing methods like Successor Features (SFs) offer some generalization, they often assume fixed transition dynamics or require re-learning for new tasks. This paper aims to develop a unified framework that allows for generalization across both reward and transition function variations, effectively combining the strengths of model-free and model-based RL.

Methods

The core contribution is the introduction of Linear Successor Feature Models (LSFMs). LSFMs learn state representations that are predictive of future expected reward outcomes, a characteristic of model-based RL. This is achieved by connecting Successor Features (SFs) to reward-predictive state representations.

Key methodological aspects include:

  • Reward-Predictive State Representations: These representations are designed to support accurate predictions of future expected reward outcomes, independent of a specific policy.
  • Linear Successor Feature Models (LSFMs): LSFMs learn a state representation that approximates SFs. This allows them to connect SFs to model-based RL.
  • Connection to Linear Action Models (LAMs): The paper shows that LSFMs are equivalent to LAMs, which are a form of model-based RL.
  • Bisimulation Relations: The paper leverages the concept of bisimulation to define conditions under which state representations generalize across states.
  • Loss Function for Learning: A specific loss function is designed to learn LSFMs by minimizing reward prediction errors and SF prediction errors.

Datasets

  • Column World: Used for illustrating state representations and the learning process of LAMs and LSFMs.
  • Puddle-World Task: Used for evaluating the performance of LAMs and LSFMs in learning approximate reward-predictive state representations.
  • Grid-World Tasks (Task A and Task B): Used for transfer experiments to demonstrate how reward-predictive state representations generalize across different transition and reward functions.
  • Combination Lock Tasks (Training, Test Lock 1, Test Lock 2): Used to illustrate how reward-predictive representations encode task-relevant state information and generalize across tasks with modified dynamics and rewards.

Key Insights and Ideas

  • SFs as a Bridge: Successor Features, when learned in a specific way (LSFMs), can effectively bridge the gap between model-free and model-based RL.
  • Reward-Predictive Generalization: State representations that are predictive of future reward sequences generalize better across tasks with varying transitions and rewards compared to value-predictive representations.
  • Overfitting of Value-Predictive Representations: Value-predictive state representations tend to "overfit" to the optimal policy of a specific task, hindering generalization to tasks with different optimal policies.
  • LSFMs Enable Model-Based RL: By learning reward-predictive state representations, LSFMs enable agents to perform model-based RL, predicting future outcomes for arbitrary action sequences.
  • Bisimulation is Key for Generalization: The paper highlights the importance of bisimulation relations for ensuring that state representations generalize across states.

Key Results

  • LSFMs learn state representations that are predictive of future expected reward outcomes, enabling generalization across variations in transitions and rewards.
  • Experiments on the Puddle-World task showed that LSFMs and LAMs learn approximate reward-predictive state representations, producing significantly lower prediction errors than random initializations.
  • Transfer experiments demonstrated that reward-predictive state representations (learned via LSFMs) enable agents to learn optimal policies more efficiently on new tasks compared to value-predictive representations, especially with limited data.
  • The Combination Lock experiments showed that reward-predictive representations capture task-relevant information for predicting reward sequences, while value-predictive representations can lead to negative transfer.

Limitations and Future Work

  • Linearity Assumption: The current LSFM framework is linear. Extending it to non-linear latent transition and reward functions is left for future work.
  • Data Requirements: The effectiveness of learning state representations relies on having a sufficiently large and representative data set of transitions. The interaction between limited training data and generalization ability is an area for future study.
  • Efficient Exploration: Combining the presented algorithms with efficient exploration strategies to improve sample complexity and regret bounds is a direction for future research.
  • Learning Neural Networks: How to learn neural networks that map inputs to latent feature spaces predictive of future reward outcomes is beyond the scope of this article and is left for future work.

Summary updated

BibTeX
@article{lehnert2020successor,
  author = {Lucas Lehnert and Michael L. Littman},
  title = {{Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
}
Figure