Code-specific policy gradient rules for spiking neurons

Henning Sprekeler, Guillaume Hennequin, Wulfram GerstnerNeurIPS 2009 · OpenReview

TL;DR

This paper derives specialized reinforcement learning rules for spiking neurons based on the specific "neural code" they use (e.g., spike counts versus precise timing). By tailoring the learning rule to the relevant features of a spike train, the authors demonstrate how to reduce noise and potentially improve learning efficiency compared to "one-size-fits-all" approaches.

The Credit Assignment Problem in Spiking Networks

In reinforcement learning (RL), "credit assignment" is the difficult task of figuring out which action led to a reward. For spiking neurons, this is exceptionally complex. Beyond the usual questions of when an action happened (temporal credit assignment) or which neuron was responsible (spatial credit assignment), there is a third dimension: What feature of the spike pattern actually mattered?

Was the reward triggered because the neuron fired a high volume of spikes (a rate code)? Or because the first spike arrived at a very precise millisecond (a timing code)?

Most existing RL rules for spiking neurons assume the "full spike train" is important. While versatile, this general-purpose approach is inherently noisy. If only the spike count matters for a specific task, but the learning rule is trying to optimize every single microsecond of timing, the network is essentially trying to learn from noise. This paper seeks to derive learning rules that are mathematically optimized for specific neural codes.

Deriving Code-Specific Learning Rules

The authors use the Policy Gradient framework (specifically the REINFORCE algorithm by Williams, 1992) as their foundation. The core idea is to adjust neuronal parameters, like synaptic weights $\theta$, to maximize the expected reward.

The key innovation is the assumption that the reward $R$ depends only on a set of coding features $F$ (like spike count or latency) rather than the raw, uncompressed spike train $Y$.

  1. The NEF Assumption: The authors assume the probability distribution of these features belongs to the Natural Exponential Family (NEF)—a broad class of distributions including Poisson, Bernoulli, and Gaussian.
  2. Derivation: By applying the policy gradient to these distributions, they derive a characteristic learning rule: $$\partial_t \theta_i = \eta R \sum_j \frac{F_j - \mu_j}{\sigma_j^2} \partial_{\theta_i} \mu_j$$ This rule compares the actual feature value ($F_j$) to its expected value ($\mu_j$), scaled by variance.

  3. Application: They apply this framework to derive specific rules for:

  4. Spike Count Codes: Where only the total number of spikes in a window matters.
  5. First-Spike Latency Codes: Where only the timing of the very first spike matters.
  6. Full Spike Train Codes: The most general case, discretized into tiny Bernoulli bins.

Experimental Setup and Simulation Results

Rather than standard real-world datasets, the paper uses synthetic "toy" problems designed to test specific coding hypotheses:

  • 2-Armed Bandit Task: A simulated agent chooses between two actions based on the spike counts of two output neurons. Rewards are given based on the state-action pair.
  • Target Latency Task: Neurons are rewarded for firing their first spike at a specific, pre-defined time after a stimulus.
Simulation setups and results for the 2-armed bandit and target latency tasks.

The authors successfully provided closed-form learning rules for the three major coding schemes. In the 2-armed bandit task, the Spike Count Rule successfully trained the network to favor the rewarded action by suppressing the "wrong" neuron's activity. Furthermore, the Latency Rule successfully shifted the first-spike times of neurons to match arbitrary target latencies (e.g., 10ms or 30ms) with high accuracy. They also showed that when the "coding feature" is the entire spike train, their framework perfectly recovers existing state-of-the-art rules.

Key Insights

  • The Signal-to-Noise Ratio (SNR) Argument: The core realization of the paper is that general rules are inherently noisier. If a task only requires a specific spike count, a "full spike train" rule will introduce "diffusion" (noise) into the learning process by trying to account for irrelevant timing fluctuations.
  • Connection to BCM Theory: The derived spike-count rule bears a striking resemblance to the famous Bienenstock-Cooper-Munro (BCM) rule. In BCM, a "sliding threshold" determines whether a synapse strengthens or weakens; here, that threshold emerges naturally as the expected mean response ($\mu$) used to explore the reward landscape.
  • Intrinsic Plasticity: The framework isn't just for synaptic weights. The authors show it can also derive rules for "intrinsic parameters," such as a neuron's excitability or firing threshold.
  • Dual Prediction Problems: The math suggests that for a neuron to learn efficiently, it needs to solve two internal prediction problems: predicting the reward it expects to get (the "critic"), and predicting its own average response to a stimulus ($\mu$).

Limitations and Future Directions

Interestingly, in their simulations, the code-specific rules didn't significantly outperform the general rules. The authors attribute this to other sources of noise in the system (a "noise bottleneck") that masked the theoretical SNR advantage.

Additionally, the rules require the neuron to "know" its own average response ($\mu$) to a specific input. In a real biological brain, it remains unclear how a single neuron calculates this, suggesting it might require top-down signals or population-level averages. A fascinating future direction is "metalearning": investigating whether a biological system could dynamically switch between these learning rules depending on which neural code is currently most effective for a given task.

Summary updated

Abstract

Although it is widely believed that reinforcement learning is a suitable tool for describing behavioral learning, the mechanisms by which it can be implemented in networks of spiking neurons are not fully understood. Here, we show that different learning rules emerge from a policy gradient approach depending on which features of the spike trains are assumed to influence the reward signals, i.e., depending on which neural code is in effect. We use the framework of Williams (1992) to derive learning rules for arbitrary neural codes. For illustration, we present policy-gradient rules for three different example codes - a spike count code, a spike timing code and the most general ``full spike train code - and test them on simple model problems. In addition to classical synaptic learning, we derive learning rules for intrinsic parameters that control the excitability of the neuron. The spike count learning rule has structural similarities with established Bienenstock-Cooper-Munro rules. If the distribution of the relevant spike train features belongs to the natural exponential family, the learning rules have a characteristic shape that raises interesting prediction problems.
BibTeX
@inproceedings{sprekeler2009code,
  author = {Henning Sprekeler and Guillaume Hennequin and Wulfram Gerstner},
  title = {{Code-specific policy gradient rules for spiking neurons}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=rybUsP-Obr},
}
Figure