Abstract
Reinforcement learning is the process by which an agent learns to predict long-term future reward. We now understand a great deal about the brain's reinforcement learning algorithms, but we know considerably less about the representations of states and actions over which these algorithms operate. A useful starting point is asking what kinds of representations we would want the brain to have, given the constraints on its computational architecture. Following this logic leads to the idea of the successor representation, which encodes states of the environment in terms of their predictive relationships with other states. Recent behavioral and neural studies have provided evidence for the successor representation, and computational studies have explored ways to extend the original idea. This paper reviews progress on these fronts, organizing them within a broader framework for understanding how the brain negotiates tradeoffs between efficiency and flexibility for reinforcement learning.
Introduction: The Efficiency-Flexibility Tradeoff
In Reinforcement Learning (RL), agents face a fundamental challenge: balancing the speed of learning with the flexibility to adapt to changes. This is often framed as a tradeoff between two systems:
- Model-Based RL: Highly flexible, it learns a full map of the environment and plans by simulating future paths. However, this is computationally expensive and slow.
- Model-Free RL: Computationally cheap, it learns the value of states based on past rewards (forming habits). However, it is brittle; changes in the environment require relearning from scratch.
The Successor Representation (SR) acts as a predictive middle ground, offering a way to represent the world that allows for rapid adaptation without the overhead of full-scale mental simulation.
Computational Logic of the Successor Representation
Originally proposed by Peter Dayan in 1993, the SR factors the value of a state into two components: the immediate Reward Function ($R$) and the Successor Representation ($M$). The matrix $M$ encodes the expected future occupancy of all other states, given the current state.
Unlike model-free methods that learn a single scalar value, the SR learns a vector of predictions. It utilizes Temporal Difference (TD) learning—the same algorithm used for reward prediction—but applies it to state transitions. This allows for "latent learning," where an agent can update its predictive map even in the absence of rewards.
Key Insights and Behavioral Evidence
A primary advantage of the SR is the "zero-shot" update. Because it separates "where you go" from "what you get," if a reward at a specific location changes, the agent can instantly update the value of all states leading to that location.
Research into human behavior supports this model. In revaluation tasks, humans adapt almost instantly to reward changes but are significantly slower to adapt to changes in the environment's layout (transitions). This "asymmetric flexibility" is a signature prediction of the SR. Furthermore, the SR provides a formal mechanism for how animals learn the structure of a maze without food, facilitating faster goal-finding once rewards are introduced.
Neural Substrates: Hippocampus and Dopamine
The paper argues that the hippocampus functions as a predictive map. Rather than simple spatial markers, hippocampal place cell firing fields may represent the SR, reflecting where an animal is likely to go next. This explains why place fields skew in the direction of travel and cluster around rewarded goals.
Additionally, Gershman suggests that dopamine might carry a vector-valued error signal used to update the entire SR matrix, moving beyond the traditional view of dopamine as a scalar reward prediction error.
Limitations and Future Directions
While the SR is a compelling model, several questions remain. The idea of vector-valued dopamine is speculative and requires further empirical validation. Additionally, the current model assumes perfect state knowledge, whereas real-world sensory input is noisy. Integrating the SR with state inference and understanding how the brain switches between SR, model-based, and model-free systems are key areas for future research.
Summary updated