Abstract
To solve complex navigation tasks, autonomous agents such as rats or mobile robots often employ spatial representations. These "maps" can be used for localisation and navigation. We propose a model for spatial learning and navigation based on reinforcement learning. The state space is represented by a population of hippocampal place cells whereas a large number of locomotor neurons in nucleus accumbens forms the action space. Using overlapping receptive fields for both populations, state/action mappings rapidly generalise during learning. The population vector allows a continuous interpretation of both state and action spaces. An eligibility trace is used to propagate reward information back in time. It enables the modification of behaviours for recent states. We propose a biologically plausible mechanism for this trace of events where spike timing dependent plasticity triggers the storing of recent state/action pairs. These pairs, however, are forgotten in the absence of a reward-related signal such as dopamine. The model is validated on a simulated robot platform.
Introduction
Bridging the gap between abstract reinforcement learning (RL) and neurobiology, this research proposes a model for navigation in continuous state and action spaces. Traditional RL often relies on discrete grids and a limited set of actions (e.g., cardinal directions). However, biological navigation occurs in a continuous world, presenting two major challenges: generalization across similar states and the credit assignment problem. This model addresses these by using population coding inspired by hippocampal place cells and locomotor neurons in the nucleus accumbens, achieving rapid generalization and providing a biologically plausible mechanism for eligibility traces via dopamine-modulated synaptic plasticity.
Methods: Neural Architecture and Population Coding
The architecture utilizes a neural-inspired RL framework to handle continuous environments.
- Continuous State Space: Represented by a population of Hippocampal Place Cells (PCs). These cells have overlapping circular receptive fields; as the agent moves, the "population vector" of active cells provides a precise, continuous coordinate.
- Continuous Action Space: Represented by Action Cells (ACs) in the nucleus accumbens. Each cell codes for a specific direction. The final movement direction is determined by the population vector of these cells.
- Biological Eligibility Trace: The core novelty is the proposed mechanism for the eligibility trace. The authors suggest that Spike-Timing Dependent Plasticity (STDP) triggers a change in the synapse that is normally vulnerable to "depotentiation" (fading away). If a reward-related dopamine signal arrives, it blocks this depotentiation, effectively "locking in" the memory of the recent state-action pair.
Key Insights and Results
The model was evaluated using a simulated robot platform in a square 80x80 cm environment. The task was to find a hidden goal location from various starting points.
Rapid Generalization and Learning
The simulated agent successfully learns to navigate to the goal in approximately 10 trials. A striking insight is that a functional navigation map (knowing where to go) emerges much faster than the convergence of the underlying Q-values. Qualitative analysis shows that after just 10 trials, the population vectors across the entire environment point consistently toward the goal, even though the Q-values continue to refine for up to 100 trials.
Population Coding as Function Approximation
Instead of using complex mathematical function approximators, the model uses overlapping radial basis functions (place cells). This naturally allows for linear interpolation between states and actions, leading to significantly faster learning. The model demonstrates that the physical "tuning width" (the size of the place cell fields) determines the resolution and generalization capabilities, rather than just the raw number of neurons.
Discussion and Future Work
While the model provides a biologically plausible bridge between RL and neural dynamics, several areas remain for exploration. The authors acknowledge that how dopaminergic neurons process information across different timesteps (the "delay line" problem) is not fully addressed. Furthermore, the exact biochemical time constants required for efficient learning in larger, more complex environments need further study. Finally, while the current model uses a modified $\epsilon$-greedy strategy, future work could investigate more sophisticated, biologically grounded exploration mechanisms.
Summary updated