Reinforcement Learning in Continuous State and Action Space

Thomas Strösslin, Wulfram GerstnerInfoscience 2003

Abstract

To solve complex navigation tasks, autonomous agents such as rats or mobile robots often employ spatial representations. These "maps" can be used for localisation and navigation. We propose a model for spatial learning and navigation based on reinforcement learning. The state space is represented by a population of hippocampal place cells whereas a large number of locomotor neurons in nucleus accumbens forms the action space. Using overlapping receptive fields for both populations, state/action mappings rapidly generalise during learning. The population vector allows a continuous interpretation of both state and action spaces. An eligibility trace is used to propagate reward information back in time. It enables the modification of behaviours for recent states. We propose a biologically plausible mechanism for this trace of events where spike timing dependent plasticity triggers the storing of recent state/action pairs. These pairs, however, are forgotten in the absence of a reward-related signal such as dopamine. The model is validated on a simulated robot platform.

Introduction

Bridging the gap between abstract reinforcement learning (RL) and neurobiology, this research proposes a model for navigation in continuous state and action spaces. Traditional RL often relies on discrete grids and a limited set of actions (e.g., cardinal directions). However, biological navigation occurs in a continuous world, presenting two major challenges: generalization across similar states and the credit assignment problem. This model addresses these by using population coding inspired by hippocampal place cells and locomotor neurons in the nucleus accumbens, achieving rapid generalization and providing a biologically plausible mechanism for eligibility traces via dopamine-modulated synaptic plasticity.

Methods: Neural Architecture and Population Coding

The architecture utilizes a neural-inspired RL framework to handle continuous environments.

The model architecture features Place Cells (PC) representing the state space and Action Cells (AC) representing the action space, with weights $w_{ij}$ connecting them.
  • Continuous State Space: Represented by a population of Hippocampal Place Cells (PCs). These cells have overlapping circular receptive fields; as the agent moves, the "population vector" of active cells provides a precise, continuous coordinate.
  • Continuous Action Space: Represented by Action Cells (ACs) in the nucleus accumbens. Each cell codes for a specific direction. The final movement direction is determined by the population vector of these cells.
  • Biological Eligibility Trace: The core novelty is the proposed mechanism for the eligibility trace. The authors suggest that Spike-Timing Dependent Plasticity (STDP) triggers a change in the synapse that is normally vulnerable to "depotentiation" (fading away). If a reward-related dopamine signal arrives, it blocks this depotentiation, effectively "locking in" the memory of the recent state-action pair.

Key Insights and Results

The model was evaluated using a simulated robot platform in a square 80x80 cm environment. The task was to find a hidden goal location from various starting points.

After learning, the population vectors across the environment point consistently toward the goal (red circle), demonstrating a functional navigation map.

Rapid Generalization and Learning

The simulated agent successfully learns to navigate to the goal in approximately 10 trials. A striking insight is that a functional navigation map (knowing where to go) emerges much faster than the convergence of the underlying Q-values. Qualitative analysis shows that after just 10 trials, the population vectors across the entire environment point consistently toward the goal, even though the Q-values continue to refine for up to 100 trials.

Population Coding as Function Approximation

Instead of using complex mathematical function approximators, the model uses overlapping radial basis functions (place cells). This naturally allows for linear interpolation between states and actions, leading to significantly faster learning. The model demonstrates that the physical "tuning width" (the size of the place cell fields) determines the resolution and generalization capabilities, rather than just the raw number of neurons.

Discussion and Future Work

While the model provides a biologically plausible bridge between RL and neural dynamics, several areas remain for exploration. The authors acknowledge that how dopaminergic neurons process information across different timesteps (the "delay line" problem) is not fully addressed. Furthermore, the exact biochemical time constants required for efficient learning in larger, more complex environments need further study. Finally, while the current model uses a modified $\epsilon$-greedy strategy, future work could investigate more sophisticated, biologically grounded exploration mechanisms.

Summary updated

Abstract

To solve complex navigation tasks, autonomous agents such as rats or mobile robots often employ spatial representations.These "maps" can be used for localisation and navigation.We propose a model for spatial learning and navigation based on reinforcement learning.The state space is represented by a population of hippocampal place cells whereas a large number of locomotor neurons in nucleus accumbens forms the action space.Using overlapping receptive fields for both populations, state/action mappings rapidly generalise during learning.The population vector allows a continuous interpretation of both state and action spaces.An eligibility trace is used to propagate reward information back in time.It enables the modification of behaviours for recent states.We propose a biologically plausible mechanism for this trace of events where spike timing dependent plasticity triggers the storing of recent state/action pairs.These pairs, however, are forgotten in the absence of a reward-related signal such as dopamine.The model is validated on a simulated robot platform.
BibTeX
@inproceedings{strosslin2003reinforcement,
  author = {Thomas Strösslin and Wulfram Gerstner},
  title = {{Reinforcement Learning in Continuous State and Action Space}},
  booktitle = {Infoscience},
  year = {2003},
}
Figure