Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation

Craig Sherstan, Marlos C. Machado, Patrick M. Pilarski

TL;DR

This paper demonstrates how the Successor Representation (SR) can significantly speed up learning in robots that build their own "knowledge systems" through General Value Functions (GVFs). By decoupling the environment's dynamics from specific prediction targets, the authors show that a robot can learn new predictive "questions" much faster and with fewer samples than traditional reinforcement learning methods.

What Problem Does This Paper Address?

In the quest for Artificial General Intelligence, researchers want robots to engage in knowledge construction: building their own models of the world directly from experience. A popular way to do this is through GVFs—predictive "questions" like, "How much heat will my motor generate over the next 10 seconds if I keep moving forward?"

However, real-world environments are complex. A robot might need to track thousands of these GVFs simultaneously. Standard Temporal Difference (TD) learning methods are often sample-inefficient in these settings; every time a new GVF is added to the system, the agent essentially starts learning from scratch. This paper addresses the challenge of how to make this incremental learning process faster and more efficient.

Methods

The authors leverage a mathematical "trick" called the Successor Representation (SR).

Standard RL methods try to learn the relationship between a state and a long-term outcome all at once. The SR instead factorizes the problem into two parts:

  1. The Dynamics: Where is the agent likely to go in the future under its current policy? (The SR)
  2. The Target: What is the immediate value of the signal I’m interested in at each state? (The Cumulant)

By learning the SR separately, the robot captures the "flow" of the environment. When a new GVF is added, the robot doesn't need to re-learn how the world works; it only needs to learn the immediate mapping of the new signal. The authors extend this to Successor Features, allowing the method to work with high-dimensional data using linear function approximation (specifically tile coding).

Datasets

The approach was evaluated in two distinct environments:

  • Dayan’s Grid World: A simulated 2D navigation task used for rigorous analysis. The authors generated 50 random signal primitives (sine waves, square waves, etc.) for the agent to predict.
  • Physical Robot Arm: A real-world setup where a human operator used a joystick to guide a robot arm through a wire maze. The dataset included sensor readings for joint position, speed, and motor current over approximately 50 circuits (12 minutes of data).

Key Insights and Ideas

  • Decoupling is Reusable: The environment's dynamics (the SR) are independent of what is being predicted. Once a robot understands how it moves through a maze, that knowledge can be reused for any new sensor it starts tracking.
  • The "Warm Start" Advantage: In a continual learning setting, as the SR becomes more accurate, the learning curve for every subsequently added GVF becomes much steeper. The robot effectively gets a "head start" on every new task.
  • Computational Scaling: The paper notes a significant benefit for massive systems. If you have $f$ different timescales and $h$ different signals, a standard system needs $f \times h$ predictors. Using the SR approach, you only need $f + h$ components, drastically reducing the number of weights the robot needs to update every second.

Key Results

  • Grid World: The SR-based method outperformed direct TD learning in the vast majority of tested signals. The advantage was most pronounced as the "look-ahead" (discount factor $\gamma$) increased.
  • Robot Arm: On real-world, noisy data, the SR-based predictors learned newly added signals (like joint speed or current) significantly faster than the direct prediction baseline.
  • Sample Efficiency: The authors showed that while the SR itself takes time to learn (initially showing higher error), once it stabilizes, the error for new GVFs drops much faster than in standard systems.

Limitations and Future Work

  • Signal-to-Noise Sensitivity: The authors noted that the SR-based approach might be more susceptible to noise when the target signal has a very small magnitude.
  • Fixed Policies: This study focused on learning predictions under a single, stable policy. A major future direction is extending this to settings where the robot's behavior (policy) is also changing over time.
  • Deep Learning Integration: While this paper used linear function approximation, the authors suggest that combining SR-based predictions with deep feature learning could be a "very powerful tool" for complex, unstructured robotic domains.

Summary updated

Abstract

We propose using the Successor Representation (SR) to accelerate learning in a constructive knowledge system based on General Value Functions (GVFs). In real-world settings, like robotics for unstructured and dynamic environments, it is impossible to model all meaningful aspects of a system and its environment by hand. Instead, robots must learn and adapt to changes in their environment and task, incrementally constructing models from their own experience. GVFs, taken from the field of reinforcement learning (RL), are a way of modeling the world as predictive questions. One approach to such models proposes a massive network of interconnected and interdependent GVFs, which are incrementally added over time. It is reasonable to expect that new, incrementally added predictions can be learned more swiftly if the learning process leverages knowledge gained from past experience. The SR provides a means of capturing regularities that can be reused across multiple GVFs by separating the dynamics of the world from the prediction targets. As a primary contribution of this work, we show that using the SR can improve sample efficiency and learning speed of GVFs in a continual learning setting where new predictions are incrementally added and learned over time. We analyze our approach in a grid-world and then demonstrate its potential on data from a physical robot arm.
BibTeX
@inproceedings{sherstan2018accelerating,
  author = {Craig Sherstan and Marlos C. Machado and Patrick M. Pilarski},
  title = {{Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=hMXEQNIJgr},
}

Loading related content...

Figure