Abstract
Synaptic plasticity, gated by reward-encoding neuromodulators, is considered the physiological basis of learning. This study investigates reward-modulated synaptic learning rules for spiking neurons in a continuous-space navigation task inspired by the Morris Water Maze. We compare an optimal rule derived from policy gradient (PG) methods with a reward-modulated Hebbian learning rule. In a network architecture combining feedforward input with lateral "Mexican-hat" connectivity, we demonstrate that standard PG methods fail to solve the task. Conversely, a Hebbian-biased variant successfully learns the navigation task within 20 trials, aligning with experimental rodent data. This work bridges formal reinforcement learning theory with synaptic-level biophysics.
Introduction
A fundamental challenge in neuroscience is the "distal reward problem": how can individual synapses determine their contribution to a reward that occurs significantly later in time? While machine learning provides mathematically optimal solutions through Policy Gradient (PG) methods, mapping these onto the stochastic, continuous-time dynamics of biological spiking neurons remains difficult. Most reinforcement learning (RL) models rely on discrete states, yet animals operate in continuous environments. This study explores how biological networks, which often represent decisions through "activity bumps" formed by lateral inhibition, implement effective learning rules.
Network Architecture and Action Selection
The model employs a two-layer spiking neural network:
- Input Layer (Place Cells): 100 Poisson neurons with Gaussian receptive fields representing the agent's 2D location.
- Output Layer (Action Cells): 360 neurons arranged in a ring, representing swimming directions.
Crucially, the action layer utilizes "Mexican-hat" connectivity—local excitation and long-range inhibition. This structural feature forces the network to form a single, localized "activity bump" of high firing rates, which determines the agent's current direction. Actions are read out at theta frequency, mimicking biological rhythms.
The Three-Factor Learning Rule
The synaptic update rule is a three-factor model where changes depend on presynaptic spike timing, postsynaptic membrane potential/spikes, and a global reward signal (representing neuromodulators like dopamine). The authors introduce a parameter, $\tau_c$, to interpolate between a pure policy gradient rule ($\tau_c = 0$) and a Hebbian-biased rule ($\tau_c > 0$).
The Hebbian component ($\tau_c = 5$ ms) reinforces synapses of neurons that are highly active during a rewarded trial. Unlike the pure PG rule, which correlates rewards with stochastic fluctuations around the mean firing rate, the Hebbian-biased rule rewards the "activity bump" itself. As shown in the figure above, the Hebbian bias leads to a more significant accumulation of the eligibility trace during periods of high activity and elevated membrane potential.
Results: Why Policy Gradients Fail
In the Morris Water Maze simulation, the agent must find a hidden platform in a 100x100 cm pool. The study revealed a striking divergence in performance:
- Policy Gradient Failure: When lateral connections are strong enough to produce stable activity bumps, the pure PG rule ($\tau_c = 0$) fails to solve the task. The PG method attempts to learn from tiny, non-causal fluctuations within the bump, which are overshadowed by the dominant lateral dynamics.
- Hebbian Success: The Hebbian-biased model ($\tau_c = 5$ ms) consistently solves the maze within 20 trials, matching the learning curves observed in real rodents. By reinforcing the entire bump, the rule becomes robust to noise and effectively maps spatial locations to successful actions.
Discussion and Future Directions
This research demonstrates that "optimal" machine learning algorithms may be ill-suited for the specific structural constraints of biological networks, such as lateral inhibition. The Hebbian-biased rule provides a biologically plausible alternative that is computationally efficient and independent of population size.
Future work aims to integrate this model with Temporal Difference (TD) learning to handle larger environments where rewards are even more distal. Additionally, incorporating plasticity into the place cell representations themselves would provide a more holistic model of hippocampal-striatal learning. This approach marks a significant step toward a unified theory of reinforcement learning that spans from behavioral outcomes down to single-synapse biophysics.
Summary updated