Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation

Wenjie Wei, Malu Zhang, Hong Qu, Ammar Belatreche

TL;DR

This paper introduces DTA-TTFS, a novel direct training algorithm for Time-To-First-Spike (TTFS) Spiking Neural Networks (SNNs). By proposing a dynamic firing threshold (DFT) mechanism, they overcome key limitations of existing TTFS-SNN training, achieving state-of-the-art accuracy with improved sparsity and energy efficiency.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) hold immense promise for energy-efficient and biologically plausible computation, especially when using temporal coding schemes like Time-To-First-Spike (TTFS). TTFS encodes information in the precise timing of the first spike, leading to sparse activity and reduced power consumption compared to traditional rate-based methods. However, training deep TTFS-SNNs effectively has been a significant challenge. Existing methods struggle with issues like "over-sparsity" of spikes, where too few neurons fire, hindering learning, and the computational complexity of identifying the "causal set" (the set of input spikes that contribute to an output spike). These problems limit the scalability and practical application of TTFS-SNNs.

Methods

The core innovation of this paper is the Dynamic Firing Threshold (DFT) mechanism. Unlike a fixed threshold, the DFT is layer-dependent and varies over time. Specifically, it's infinite before a designated time window for a layer and then linearly decreases within that window. This design has two key benefits:

  1. Addressing Over-sparsity: The linearly decreasing threshold keeps neurons active for longer within their designated time window, preventing premature deactivation and ensuring more neurons contribute to learning.
  2. Simplifying Causal Set Identification: The layer-dependent, non-overlapping spike time windows ensure that all active presynaptic neurons within the previous layer's window are inherently part of the postsynaptic neuron's "causal set." This eliminates the need for computationally expensive sorting and iteration to find the causal set.

Building upon the DFT, the authors propose DTA-TTFS (Direct Training Algorithm for TTFS-based SNNs). This algorithm leverages the DFT and an event-driven processing approach to train TTFS-SNNs directly. It uses a cross-entropy loss function based on negative spike times and employs gradient backpropagation, calculating derivatives of spike times with respect to synaptic weights and input spike times. The use of the ReL-PSP neuron model and the DFT allows for analytical solutions, enabling efficient training without simulating long time steps, thus facilitating the use of standard GPU acceleration.

Datasets

The proposed DTA-TTFS method was evaluated on the following standard image classification benchmark datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of 10 classes of small images.
  • CIFAR-100: A dataset of 100 classes of small images.

The paper does not introduce new benchmarks.

Key Insights and Ideas

  • DFT as a Game Changer: The dynamic firing threshold is a surprisingly simple yet effective mechanism to tackle fundamental issues in TTFS-SNN training. It elegantly addresses both the over-sparsity problem and the computational burden of causal set identification.
  • Layer-Dependent Spike Timing: The DFT enforces a structured, non-overlapping spike timing across layers. This "layer-dependent firing feature" is crucial for efficient learning and prevents spikes from propagating indefinitely.
  • Bridging the Gap to Efficient Training: By enabling analytical solutions and avoiding long time-step simulations, the DFT-based approach allows for the use of standard GPU-accelerated training pipelines, making deep SNN training more accessible.
  • Balancing Act: The research highlights the critical need to balance sparsity (for energy efficiency) with model performance. The DTA-TTFS method demonstrates a successful trade-off, achieving high accuracy while maintaining competitive sparsity.

Key Results

  • State-of-the-Art Accuracy: DTA-TTFS achieved excellent performance on image classification tasks:
  • MNIST: 99.4% accuracy.
  • CIFAR-10: 93.05% accuracy.
  • CIFAR-100: 69.66% accuracy. These results outperform previous direct learning algorithms for TTFS-SNNs.

  • Improved Sparsity: For MNIST, DTA-TTFS achieved a sparsity of 0.3913, significantly reducing it from 0.6614 reported in prior work while maintaining the same accuracy. For CIFAR datasets, it maintained competitive sparsity.

  • Energy Efficiency: Theoretical analysis and hardware simulation on the CanMore neuromorphic simulator demonstrated significant energy savings. On CIFAR-10, DTA-TTFS achieved 90.67% accuracy with only 50.36 µJ/frame, a 69.2% reduction compared to a previous ANN-to-SNN conversion method (163.65 µJ/frame).

Limitations and Future Work

  • Limitations:
    • While the paper demonstrates strong performance, the "over-sparsity" issue is still a concern in TTFS-SNNs, and the DFT aims to mitigate rather than completely eliminate it.
  • The paper focuses on image classification; its performance on other complex tasks like natural language processing or reinforcement learning is not explored.
  • Future Work:
    • The authors plan to extend DTA-TTFS to train more advanced network architectures, specifically mentioning SNN-based Attention mechanisms and Transformer architectures.
  • Further exploration of the DFT's impact on different SNN architectures and tasks would be valuable.

Summary updated

Abstract

Spiking Neural Networks (SNNs) offer a highly promising computing paradigm due to their biological plausibility, exceptional spatiotemporal information processing capability and low power consumption. As a temporal encoding scheme for SNNs, Time-To-First-Spike (TTFS) encodes information using the timing of a single spike, which allows spiking neurons to transmit information through sparse spike trains and results in lower power consumption and higher computational efficiency compared to traditional rate-based encoding counterparts. However, despite the advantages of the TTFS encoding scheme, the effective and efficient training of TTFS-based deep SNNs remains a significant and open research problem. In this work, we first examine the factors underlying the limitations of applying existing TTFS-based learning algorithms to deep SNNs. Specifically, we investigate issues related to over-sparsity of spikes and the complexity of finding the ‘causal set’. We then propose a simple yet efficient dynamic firing threshold (DFT) mechanism for spiking neurons to address these issues. Building upon the proposed DFT mechanism, we further introduce a novel direct training algorithm for TTFS-based deep SNNs, called DTA-TTFS. This method utilizes event-driven processing and spike timing to enable efficient learning of deep SNNs. The proposed training method was validated on the image classification task and experimental results clearly demonstrate that our proposed method achieves state-of-the-art accuracy in comparison to existing TTFS-based learning algorithms, while maintaining high levels of sparsity and energy efficiency on neuromorphic inference accelerator.
BibTeX
@inproceedings{wei2024temporal,
  author = {Wenjie Wei and Malu Zhang and Hong Qu and Ammar Belatreche},
  title = {{Temporal-coded spiking neural networks with dynamic firing threshold: Learning with event-driven backpropagation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=1jrLAqqK61},
}
Figure