Stochastic variational learning in recurrent spiking networks

Danilo Jimenez Rezende, Wulfram GerstnerFrontiers Comput. Neurosci 2014 · DOI · OpenReview

Introduction

Learning in recurrent spiking neural networks (RSNNs) presents a significant challenge in computational neuroscience, particularly when hidden neurons—those not directly receiving sensory input—are involved. Traditional machine learning methods like Backpropagation Through Time (BPTT) are biologically implausible as they require non-local information and backward-in-time signaling. Conversely, standard biological rules like Spike-Timing Dependent Plasticity (STDP) often fail to capture long-term temporal structures or the "hidden causes" of sensory data.

Network architecture showing observed neurons (V) and target neurons.

Rezende and Gerstner (2014) address this by deriving a biologically plausible learning rule that combines variational inference with reinforcement learning principles. Their approach allows RSNNs to learn complex spatio-temporal patterns through a local STDP rule modulated by a global "novelty" signal.

Methods: Variational Learning in Spiking Networks

The authors treat the spiking network as a generative model using a Spike Response Model (SRM), where neurons fire stochastically based on their membrane potential. To handle the mathematical intractability of calculating the exact "cause" of a spike train, they employ variational learning.

Graphical representation of the generative model (M) and the recognition network (Q) interacting with hidden (H) and visible (V) neurons.

The framework utilizes an auxiliary "recognition" network (the Q-network) to approximate the posterior distribution of hidden neuron activity. By minimizing the Helmholtz Free Energy, the authors derive a three-factor learning rule where synaptic changes depend on:

  1. Pre-synaptic activity: A trace of past spikes.
  2. Post-synaptic state: Voltage and timing.
  3. Global Novelty Signal ($e_N$): A neuromodulator-like signal indicating how "surprising" the input is relative to the network's internal model.

To stabilize learning in large stochastic networks, they introduced a "baseline removal" trick, subtracting a moving average of the free energy to reduce gradient noise.

Experimental Evaluation

The model was tested on two primary tasks:

  • Stair Patterns: A sequence of spike bursts testing the network's ability to learn transition probabilities and long-term dependencies.
  • Virtual Maze Task: A simulated rat navigating a 16-room maze, where each room is represented by an MNIST digit. The network learned the maze topology based on sequences of these images.

Key Results

  • Necessity of Hidden Neurons: Networks without hidden neurons failed to learn complex patterns, whereas those with hidden neurons captured the data distribution effectively.
  • Computational Efficiency: The variance-reduced learning rule was over 100 times faster and more stable than naive stochastic gradient descent.
  • Novelty Detection: In the maze task, a change in layout triggered a massive burst in the global error signal ($e_N$), providing a testable prediction for biological neuromodulator activity (e.g., Dopamine or Acetylcholine).

Discussion and Insights

A central insight of this work is the role of "surprise" as a driver for plasticity. The global signal $e_N$ acts as a gate; if the network predicts the input perfectly, no learning occurs. Furthermore, the model suggests a functional duality in synapses, potentially managing weights for both generation ($M$) and inference ($Q$).

While the theory provides a robust bridge between Bayesian inference and biological plasticity, scaling to millions of neurons remains a challenge due to increasing noise. Future work involves experimental validation of the predicted novelty signals in animal models and exploring the biological implementation of the dual $M$ and $Q$ weights, perhaps through distinct neuromodulatory states like wakefulness and sleep.

Reference: Rezende, D. J., & Gerstner, W. (2014). Stochastic variational learning in recurrent spiking networks. Frontiers in Computational Neuroscience, 8, 38.

Summary updated

Abstract

The ability to learn and perform statistical inference with biologically plausible recurrent networks of spiking neurons is an important step toward understanding perception and reasoning. Here we derive and investigate a new learning rule for recurrent spiking networks with hidden neurons, combining principles from variational learning and reinforcement learning. Our network defines a generative model over spike train histories and the derived learning rule has the form of a local Spike Timing Dependent Plasticity rule modulated by global factors (neuromodulators) conveying information about “novelty” on a statistically rigorous ground. Simulations show that our model is able to learn both stationary and non-stationary patterns of spike trains. We also propose one experiment that could potentially be performed with animals in order to test the dynamics of the predicted novelty signal.
BibTeX
@article{rezende2014stochastic,
  author = {Danilo Jimenez Rezende and Wulfram Gerstner},
  title = {{Stochastic variational learning in recurrent spiking networks}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2014},
  doi = {10.3389/fncom.2014.00038},
}
Figure