Abstract
Here we propose using the successor representation (SR) to accelerate learning in a constructive knowledge system based on general value functions (GVFs). In real-world settings like robotics for unstructured and dynamic environments, it is infeasible to model all meaningful aspects of a system and its environment by hand due to both complexity and size. Instead, robots must be capable of learning and adapting to changes in their environment and task, incrementally constructing models from their own experience. GVFs, taken from the field of reinforcement learning (RL), are a way of modeling the world as predictive questions. One approach to such models proposes a massive network of interconnected and interdependent GVFs, which are incrementally added over time. It is reasonable to expect that new, incrementally added predictions can be learned more swiftly if the learning process leverages knowledge gained from past experience. The SR provides such a means of separating the dynamics of the world from the prediction targets and thus capturing regularities that can be reused across multiple GVFs. As a primary contribution of this work, we show that using SR-based predictions can improve sample efficiency and learning speed in a continual learning setting where new predictions are incrementally added and learned over time. We analyze our approach in a grid-world and then demonstrate its potential on data from a physical robot arm.
Introduction
In the quest for Artificial General Intelligence, researchers aim to enable robots to engage in autonomous knowledge construction—building models of the world directly from their own sensory experience. General Value Functions (GVFs) have emerged as a powerful framework for this, representing knowledge as predictive "questions" about future signals (e.g., "How much motor current will be used over the next 10 seconds?"). However, as these systems scale to thousands of GVFs, standard Temporal Difference (TD) learning methods become sample-inefficient. Every time a new GVF is added, the agent typically starts learning from scratch. This paper addresses how to make this incremental learning process faster and more efficient by leveraging the Successor Representation (SR).
Methodology: Decoupling Dynamics from Targets
The core of our approach is a mathematical factorization of the value function. While standard RL methods attempt to learn the relationship between a state and a long-term outcome directly, the SR factorizes the problem into two distinct parts:
- The Dynamics (The SR): This captures where the agent is likely to go in the future under its current policy, effectively modeling the "flow" of the environment.
- The Target (The Cumulant): This captures the immediate value of the specific signal of interest at each state.
By learning the SR separately, the robot captures environmental regularities that are independent of what is being predicted. When a new GVF is added, the robot does not need to re-learn how it moves through the environment; it only needs to learn the immediate mapping of the new signal to the existing SR. This provides a "warm start" for every subsequently added prediction. To handle high-dimensional data, we extend this to Successor Features, utilizing linear function approximation with tile coding.
Experimental Setup
We evaluated our approach in two distinct settings:
- Dayan’s Grid World: A simulated 2D navigation task used for rigorous analysis. The agent was tasked with predicting 50 random signal primitives (e.g., sine and square waves) to test the system's ability to handle diverse predictive targets.
- Physical Robot Arm: A real-world setup where a robot arm was guided through a wire maze by a human operator. The dataset included sensor readings for joint position, speed, and motor current collected over approximately 12 minutes of operation.
Results and Discussion
Our results demonstrate that the SR-based method significantly outperforms direct TD learning in terms of both final accuracy and learning speed. In the Grid World experiments (Figure 1), the SR-based predictors achieved lower error across a wide range of learning rates, with the advantage becoming more pronounced as the look-ahead horizon (discount factor $\gamma$) increased.
Crucially, we observed a significant improvement in sample efficiency. As shown in Figure 2, while the SR itself requires an initial period to be learned (red line), the error for new GVFs (green line) drops much faster than in standard direct prediction systems (blue line) once the SR has stabilized. This "warm start" advantage is vital for robots operating in dynamic, continual learning environments.
Beyond accuracy, the SR approach offers computational scaling benefits. In a massive system with $f$ different timescales and $h$ different signals, a standard system requires $f \times h$ predictors. By using the SR, the system only needs $f + h$ components, drastically reducing the number of weights that must be updated in real-time.
Conclusion
This work shows that using the Successor Representation can significantly accelerate learning in constructive predictive frameworks. By decoupling environmental dynamics from specific prediction targets, robots can reuse past experience to learn new predictive models with fewer samples. While the current study focused on stable policies, future work will explore extending these benefits to settings where the robot's behavior is also changing over time, potentially integrating SR-based predictions with deep feature learning for even more complex domains.
Summary updated