Direct Training via Backpropagation for Ultra-low Latency Spiking Neural Networks with Multi-threshold

Changqing Xu, Yi Liu, Yintang YangarXiv 2021 · arXiv

TL;DR

This paper introduces a novel direct training method for Spiking Neural Networks (SNNs) using backpropagation, enabling ultra-low latency (as few as 2 time steps) while maintaining high accuracy. They achieve this by proposing a multi-threshold Leaky Integrate-and-Fire (LIF) neuron model to increase information capacity per spike and developing three differentiable approximations for the non-differentiable spike activation function.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are a promising alternative to traditional Deep Neural Networks (DNNs) due to their event-driven nature, leading to significant energy efficiency and the ability to process spatio-temporal information. However, a major bottleneck for SNNs has been their high latency during both training and inference. SNNs typically require tens or even hundreds of time steps to achieve decent performance, which not only increases latency but also consumes more energy. This paper tackles the challenge of reducing this latency to an ultra-low level (1-2 time steps) without sacrificing accuracy, a goal that has been difficult to achieve with existing methods that often rely on converting pre-trained ANNs or improving coding schemes.

Methods

The core of this paper lies in two key innovations:

  • Multi-threshold LIF Neuron Model: To increase the amount of information a single spike can carry, the authors extend the standard Leaky Integrate-and-Fire (LIF) neuron model. Instead of a single firing threshold, their multi-threshold model uses multiple thresholds. This allows a neuron to emit multiple "levels" of spikes (represented by Smax in the paper) within a single time step, effectively increasing its information capacity.
  • Differentiable Approximations for Spike Activation: A fundamental challenge in training SNNs with backpropagation is the non-differentiable nature of the spike generation process. To overcome this, the authors propose three novel curves (f1, f2, f3) that approximate the derivative of the multi-threshold spike activation function. This allows for the propagation of error gradients through the network, enabling direct training. They experimentally found that f2 provided a good balance of performance and stability.

The training process involves a standard backpropagation algorithm, but adapted to work with the proposed SNN architecture and the differentiable spike approximations.

Datasets

The proposed method was evaluated on three standard image datasets:

  • MNIST: A dataset of handwritten digits.
  • FashionMNIST: A dataset of clothing items, considered more challenging than MNIST.
  • CIFAR10: A dataset of object images, requiring a deeper CNN architecture.

The authors compare their results against existing SNNs trained with backpropagation, converted SNNs, and traditional ANNs on these benchmarks.

Key Insights and Ideas

  • More Information Per Spike: The multi-threshold LIF model is a clever way to pack more information into each spike, directly addressing the latency issue by reducing the need for many temporal steps.
  • Bridging the Differentiability Gap: The proposed derivative approximations are crucial. They provide a practical way to apply gradient-based learning to SNNs, which has historically been a major hurdle.
  • Ultra-Low Latency is Achievable: The results demonstrate that high accuracy can be maintained even with extremely short inference times (2 time steps), challenging the notion that SNNs inherently require long temporal sequences.
  • Trade-offs in Derivative Approximation: While three approximations were proposed, they showed similar performance, with f2 and f3 being preferred over f1 due to potential gradient explosion issues with f1. The width of the approximation (aw) also plays a role, with excessively small values hurting performance.
  • Smax's Importance: The Smax parameter (upper limit of output spikes) is critical for information capacity. Once Smax is sufficiently large, further increases don't significantly improve accuracy, suggesting that the network can encode enough spatial information within a single spike.

Key Results

The proposed method achieved state-of-the-art or competitive results on all tested datasets with significantly reduced latency:

  • MNIST: Achieved 99.56% accuracy with only 2 time steps, outperforming previous methods that required more time steps.
  • FashionMNIST: Achieved 93.08% accuracy with 2 time steps, outperforming TSSL-BP [11] which achieved 90.19% with 5 time steps.
  • CIFAR10: Achieved 87.90% accuracy with 2 time steps, showing a 1.12% improvement over TSSL-BP [11] with fewer time steps.

The paper also demonstrated that while longer spike trains (more time steps) generally improve accuracy, the gains diminish significantly after 3 time steps, and can even lead to overfitting without additional regularization techniques.

Limitations and Future Work

  • Derivative Approximation Sensitivity: While the derivative approximations enable training, the paper notes that their exact shape is not highly sensitive to accuracy, as their primary role is to capture non-linearity. However, extreme values (like f1) can lead to gradient explosion.
  • Impact of Smax: The paper shows that Smax is important for encoding spatial information, but its impact saturates. Further research could explore optimal Smax values for different network architectures and tasks.
  • Temporal Information vs. Robustness: While ultra-low latency is achieved, the paper acknowledges that longer spike trains can carry more temporal information. The trade-off between ultra-low latency and robustness to temporal variations is an area for further investigation. The authors suggest that without additional training skills, very long spike trains might lead to overfitting.
  • Future Work: The authors implicitly suggest exploring more complex network architectures and potentially incorporating other SNN training techniques to further enhance performance and robustness while maintaining low latency. The exploration of different derivative approximation curves and their parameters also opens avenues for future optimization.

Summary updated

BibTeX
@misc{xu2021direct,
  author = {Changqing Xu and Yi Liu and Yintang Yang},
  title = {{Direct Training via Backpropagation for Ultra-low Latency Spiking Neural Networks with Multi-threshold}},
  year = {2021},
  eprint = {2112.07426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07426},
}
Figure