Training Probabilistic Spiking Neural Networks with First- To-Spike Decoding

Alireza Bagheri, Osvaldo Simeone, Bipin RajendranICASSP 2018 · DOI

TL;DR

This paper introduces a novel training method for Spiking Neural Networks (SNNs) that uses a probabilistic neuron model and a "first-to-spike" decoding strategy. This approach allows for faster and more energy-efficient classification decisions compared to traditional SNN training methods, achieving competitive accuracy with reduced computational complexity.

What Problem Does This Paper Address?

Traditional artificial neural networks (ANNs) are computationally intensive and power-hungry. Spiking Neural Networks (SNNs), inspired by biological brains, offer a path towards more energy-efficient computing by processing information through sparse, discrete "spikes." However, training SNNs effectively, especially for classification tasks, has been a significant challenge. Existing methods often rely on converting trained ANNs to SNNs or use complex, non-differentiable learning rules. Furthermore, conventional SNN decoding requires waiting for all output neurons to finish processing, which can introduce latency. This paper addresses the need for flexible and efficient learning algorithms for SNNs, specifically focusing on a probabilistic neuron model and a faster "first-to-spike" decoding mechanism.

Methods

The authors propose a two-layer SNN architecture using Generalized Linear Model (GLM) probabilistic neurons. GLM neurons are a flexible and computationally tractable model for spiking neurons, allowing for gradient-based learning. The key innovation is the training algorithm designed for a "first-to-spike" decoding rule. Instead of waiting for the total spike count from each output neuron, this rule makes a classification decision as soon as the first spike is detected from the neuron corresponding to the correct class.

The training is performed using Stochastic Gradient Descent (SGD) to maximize the likelihood that the first spike occurs at the output neuron associated with the correct class. This involves deriving gradients for the GLM neuron's parameters (synaptic kernels and feedback kernels) based on this first-to-spike criterion. The paper also contrasts this with conventional SNN training, which aims to maximize the total spike count for the correct class.

Datasets

The paper evaluates its proposed method using the standard MNIST dataset of handwritten digits. They consider classification tasks for:

  • Two digits ({5, 7})
  • Four digits ({5, 7, 1, 9})
  • All ten digits ({0, ..., 9})

Key Insights and Ideas

  • First-to-Spike Decoding for Probabilistic SNNs: This is the core contribution. Applying the first-to-spike decoding to probabilistic SNNs (specifically GLM neurons) is novel and offers a significant advantage in reducing decision latency and computational complexity during inference.
  • GLM Neurons Enable Gradient-Based Learning: The use of GLM neurons, which are inherently probabilistic and differentiable, allows for the application of standard gradient-based optimization techniques like SGD, making training more tractable than with deterministic spiking neuron models.
  • Accuracy-Complexity Trade-off: The first-to-spike decoding method demonstrates a superior accuracy-complexity trade-off. It can achieve high classification accuracy with significantly lower inference complexity compared to conventional rate-based decoding, especially when early decisions are possible.
  • Parameter Sensitivity: The results suggest that conventional decoding might require a larger number of basis functions (larger temporal receptive field) to achieve optimal accuracy, whereas first-to-spike decoding can perform well with fewer basis functions, particularly for shorter time windows.

Key Results

  • Improved Accuracy-Complexity Trade-off: Figure 5 clearly shows that first-to-spike decoding achieves comparable or better classification accuracy than conventional decoding at a fraction of the inference complexity. For example, at 98.4% accuracy, the conventional method was five times more complex.
  • Performance with Limited Time Windows: For small time windows (e.g., T=4), first-to-spike decoding achieved better accuracies than conventional decoding (Figure 4).
  • Competitive Accuracy: The proposed method achieves competitive classification accuracies on the MNIST dataset, comparable to conventional SNN decoding and even a standard ANN.

Limitations and Future Work

  • Two-Layer Network: The current study focuses on a two-layer SNN. The authors suggest that generalizing these results to multi-layer networks is a necessary next step.
  • Generalization of Probabilistic Models: While the paper uses GLM, further exploration of other probabilistic spiking neuron models and their training under first-to-spike decoding could be beneficial.
  • Real-World Neuromorphic Hardware: While SNNs are designed for neuromorphic hardware, the paper focuses on algorithmic aspects. Future work could involve implementing and evaluating these methods on actual neuromorphic chips to demonstrate practical energy savings.

Summary updated

BibTeX
@inproceedings{bagheri2018training,
  author = {Alireza Bagheri and Osvaldo Simeone and Bipin Rajendran},
  title = {{Training Probabilistic Spiking Neural Networks with First- To-Spike Decoding}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2018.8462410},
}
Figure