Correction: Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail

Eleni Vasilaki, Nicolas Frémaux, Robert Urbanczik, Walter Senn, Wulfram GerstnerPLoS 2009 · DOI

Abstract

Synaptic plasticity, gated by reward-encoding neuromodulators, is considered the physiological basis of learning. This study investigates reward-modulated synaptic learning rules for spiking neurons in a continuous-space navigation task inspired by the Morris Water Maze. We compare an optimal rule derived from policy gradient (PG) methods with a reward-modulated Hebbian learning rule. In a network architecture combining feedforward input with lateral "Mexican-hat" connectivity, we demonstrate that standard PG methods fail to solve the task. Conversely, a Hebbian-biased variant successfully learns the navigation task within 20 trials, aligning with experimental rodent data. This work bridges formal reinforcement learning theory with synaptic-level biophysics.

Introduction

A fundamental challenge in neuroscience is the "distal reward problem": how can individual synapses determine their contribution to a reward that occurs significantly later in time? While machine learning provides mathematically optimal solutions through Policy Gradient (PG) methods, mapping these onto the stochastic, continuous-time dynamics of biological spiking neurons remains difficult. Most reinforcement learning (RL) models rely on discrete states, yet animals operate in continuous environments. This study explores how biological networks, which often represent decisions through "activity bumps" formed by lateral inhibition, implement effective learning rules.

Network Architecture and Action Selection

The model employs a two-layer spiking neural network:

  1. Input Layer (Place Cells): 100 Poisson neurons with Gaussian receptive fields representing the agent's 2D location.
  2. Output Layer (Action Cells): 360 neurons arranged in a ring, representing swimming directions.

Crucially, the action layer utilizes "Mexican-hat" connectivity—local excitation and long-range inhibition. This structural feature forces the network to form a single, localized "activity bump" of high firing rates, which determines the agent's current direction. Actions are read out at theta frequency, mimicking biological rhythms.

The Three-Factor Learning Rule

The synaptic update rule is a three-factor model where changes depend on presynaptic spike timing, postsynaptic membrane potential/spikes, and a global reward signal (representing neuromodulators like dopamine). The authors introduce a parameter, $\tau_c$, to interpolate between a pure policy gradient rule ($\tau_c = 0$) and a Hebbian-biased rule ($\tau_c > 0$).

Comparison of synaptic update components for pure policy gradient ($\tau_c=0$) and Hebbian-biased ($\tau_c=5$) rules.

The Hebbian component ($\tau_c = 5$ ms) reinforces synapses of neurons that are highly active during a rewarded trial. Unlike the pure PG rule, which correlates rewards with stochastic fluctuations around the mean firing rate, the Hebbian-biased rule rewards the "activity bump" itself. As shown in the figure above, the Hebbian bias leads to a more significant accumulation of the eligibility trace during periods of high activity and elevated membrane potential.

Results: Why Policy Gradients Fail

In the Morris Water Maze simulation, the agent must find a hidden platform in a 100x100 cm pool. The study revealed a striking divergence in performance:

  • Policy Gradient Failure: When lateral connections are strong enough to produce stable activity bumps, the pure PG rule ($\tau_c = 0$) fails to solve the task. The PG method attempts to learn from tiny, non-causal fluctuations within the bump, which are overshadowed by the dominant lateral dynamics.
  • Hebbian Success: The Hebbian-biased model ($\tau_c = 5$ ms) consistently solves the maze within 20 trials, matching the learning curves observed in real rodents. By reinforcing the entire bump, the rule becomes robust to noise and effectively maps spatial locations to successful actions.

Discussion and Future Directions

This research demonstrates that "optimal" machine learning algorithms may be ill-suited for the specific structural constraints of biological networks, such as lateral inhibition. The Hebbian-biased rule provides a biologically plausible alternative that is computationally efficient and independent of population size.

Future work aims to integrate this model with Temporal Difference (TD) learning to handle larger environments where rewards are even more distal. Additionally, incorporating plasticity into the place cell representations themselves would provide a more holistic model of hippocampal-striatal learning. This approach marks a significant step toward a unified theory of reinforcement learning that spans from behavioral outcomes down to single-synapse biophysics.

Summary updated

Abstract

Changes of synaptic connections between neurons are thought to be the physiological basis of learning. These changes can be gated by neuromodulators that encode the presence of reward. We study a family of reward-modulated synaptic learning rules for spiking neurons on a learning task in continuous space inspired by the Morris Water maze. The synaptic update rule modifies the release probability of synaptic transmission and depends on the timing of presynaptic spike arrival, postsynaptic action potentials, as well as the membrane potential of the postsynaptic neuron. The family of learning rules includes an optimal rule derived from policy gradient methods as well as reward modulated Hebbian learning. The synaptic update rule is implemented in a population of spiking neurons using a network architecture that combines feedforward input with lateral connections. Actions are represented by a population of hypothetical action cells with strong mexican-hat connectivity and are read out at theta frequency. We show that in this architecture, a standard policy gradient rule fails to solve the Morris watermaze task, whereas a variant with a Hebbian bias can learn the task within 20 trials, consistent with experiments. This result does not depend on implementation details such as the size of the neuronal populations. Our theoretical approach shows how learning new behaviors can be linked to reward-modulated plasticity at the level of single synapses and makes predictions about the voltage and spike-timing dependence of synaptic plasticity and the influence of neuromodulators such as dopamine. It is an important step towards connecting formal theories of reinforcement learning with neuronal and synaptic properties.
BibTeX
@inproceedings{vasilaki2009correction,
  author = {Eleni Vasilaki and Nicolas Frémaux and Robert Urbanczik and Walter Senn and Wulfram Gerstner},
  title = {{Correction: Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail}},
  booktitle = {PLoS},
  year = {2009},
  doi = {10.1371/annotation/307ea250-3792-4ceb-b905-162d86c96baf},
}
Figure