TL;DR
By using the eigenvectors of an environment's transition matrix as a feature basis—mirroring how "grid cells" function in the brain—researchers have developed a reinforcement learning agent that adapts to new goals almost instantly. This "Spectral Successor Feature" approach eliminates the need for complex neural networks to learn environmental dynamics, reducing task adaptation to a simple linear regression on rewards.
What Problem Does This Paper Address?
In Continual Reinforcement Learning (RL), an agent must solve a sequence of tasks in the same environment. Ideally, it should reuse its knowledge of the world's "physics" (dynamics) when only the "goal" (reward) changes.
A popular framework for this is Successor Features (SFs), which decomposes the value function into two parts: a representation of future state visitations (dynamics) and a set of weights (rewards). However, standard SF methods have a major bottleneck: the successor features themselves usually have to be learned via temporal-difference (TD) learning. This process is slow, requires a replay buffer, and often needs to be restarted or heavily adapted whenever the agent's policy or the task changes.
The authors ask: Can we find a specific basis for these features that makes them "reusable" and mathematically trivial to compute, rather than learned?
Methods
The authors introduce Spectral Successor Features (Spectral-SF). Their approach is grounded in a neuroscientific hypothesis: that entorhinal grid cells in the brain represent the eigenvectors of the environment's transition structure.
- The Spectral Basis: They build a feature basis $\phi$ using the eigenvectors of the transition matrix of a fixed reference policy (typically a uniform random walk).
- Closed-Form Successors: Mathematically, if your base features are eigenvectors, the successor features $\psi$ are simply the base features multiplied by a constant: $\psi_i = \theta_i \phi_i$, where $\theta_i = 1/(1 - \gamma \lambda_i)$ and $\lambda_i$ is the eigenvalue. This means you don't need a neural network to learn the "future"; it's already encoded in the "present" features.
- Policy Separation: They decouple the reference policy (used to define the stable geometry of the world) from the acting policy (which acts greedily based on current reward estimates).
- State-Action Mapping: Crucially, they show that the basis must be built over state-action pairs $(s, a)$ rather than just states $(s)$ to properly capture how different actions lead to different outcomes in walled environments.
The paper compares four different agent architectures, summarized in the table below:
Datasets
The agents were evaluated on the Inverted-LWalls environment, a $12 \times 12$ gridworld.
- The Challenge: The agent is exposed to two different goal locations in a sequence (Goal A $\rightarrow$ Goal B $\rightarrow$ Goal A $\rightarrow$ Goal B).
- Metric: The researchers measured how well the agents retained knowledge and how quickly they adapted when the goal moved.
Key Insights and Ideas
- The "Aha!" Moment: In a spectral basis, the long-term return is predicted from the exact same features as the immediate reward. You just rescale the weights. This turns a complex RL problem into a simple linear regression.
- Geometry is Stable, Rewards are Not: By fixing the basis to a random-walk reference policy, the agent treats the "shape" of the room as a constant. When the goal moves, it doesn't have to relearn what a "wall" is; it only has to relearn where the "cheese" is.
- Neuro-Inspiration Works: The results provide computational evidence for why the brain might use grid-cell-like representations: they provide a near-optimal basis for rapid navigation and transfer learning.
Key Results
- Instant Adaptation: Both Offline and Online Spectral-SF agents reached near-maximal return almost immediately after a goal change.
- Baseline Failure: Standard DQN and SimpleSF (a state-of-the-art successor feature method) lost almost all their return every time the goal moved and had to slowly relearn the task from scratch.
- Zero Replay Buffer: Unlike DQN, Spectral-SF requires no replay buffer or bootstrapped targets for its dynamics, significantly reducing memory and computational overhead during online learning.
Limitations and Future Work
- The "Trapping" Problem: The current greedy acting policy is guaranteed to find the goal if there is only a single positive reward. However, if the environment contains "lava" (negative rewards), the agent can get trapped in sub-optimal areas because it only performs one step of policy improvement over the random walk.
- Scaling to Pixels: While the "Offline" version uses an oracle for the basis, the "Online" version currently requires knowing the cell index. The authors plan to integrate methods like ALLO (Augmented Lagrangian Laplacian Objective) to learn these spectral features directly from raw pixels.
- Environment Complexity: The current study is limited to static, deterministic gridworlds. Testing on stochastic or continuous environments remains a task for future research.
Summary updated