Gradient Scaling on Deep Spiking Neural Networks with Spike-Dependent Local Information

Seongsik Park, Jeonghee Jo, Jongkil Park, Yeonjoo Jeong, Jaewook Kim, Suyoun Lee, Joon Young Kwak, Inho Kim, Jong-Keuk Park, Kyeong Seok Lee, Gye Weon Hwang, Hyun Jae JangarXiv 2023 · arXiv

TL;DR

This paper introduces Gradient Scaling (GS), a novel training method for Deep Spiking Neural Networks (SNNs) that bridges the gap between global gradient-based optimization and biologically inspired local learning. By scaling gradients based on the temporal causality between pre- and post-synaptic spikes, the authors achieve higher classification accuracy on standard benchmarks while simultaneously reducing the number of spikes, leading to better energy efficiency.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are the "third generation" of neural networks, designed to mimic the brain's event-based communication. Unlike standard Deep Neural Networks (DNNs) that pass continuous values, SNNs communicate via discrete binary spikes, making them incredibly energy-efficient on neuromorphic hardware.

However, training them is notoriously difficult. The core operation of a spiking neuron—firing a spike when a threshold is reached—is non-differentiable, which breaks standard backpropagation. Researchers currently use Spatio-Temporal Backpropagation (STBP) with "surrogate gradients" to get around this. The problem is that these methods are essentially ported from the DNN world; they treat SNNs like discretized DNNs and ignore the rich, local temporal dynamics (the "causality") between spikes that biological brains use to learn via Spike-Timing-Dependent Plasticity (STDP).

Methods

The authors propose a hybrid approach: keep the global optimization power of STBP but modulate it with local spike information.

  1. Spike Traces: For every neuron, they maintain a "spike trace" ($x$). When a neuron fires, its trace jumps up; otherwise, it decays exponentially over time. This trace acts as a short-term memory of recent spiking activity.
  2. Causality Relationship ($R$): They define a relationship function that compares the traces of a pre-synaptic neuron (the sender) and a post-synaptic neuron (the receiver). If a pre-synaptic spike is closely followed by a post-synaptic spike, there is high causality.
  3. Gradient Scaling (GS): During the backward pass, the standard gradient calculated by STBP is scaled by this causality factor. If the spikes show a strong causal relationship, the gradient is "encouraged" (amplified). If the relationship is weak, the gradient is "hindered" (diminished).
Figure 1. Spike trace and proposed gradient scaling (GS).

This mechanism is inspired by STDP, where synaptic connections are strengthened if the input consistently triggers the output, but it is integrated directly into a modern, scalable gradient-descent framework.

Datasets

The researchers evaluated their method using two standard computer vision benchmarks:

  • CIFAR-10: A dataset of 60,000 32x32 color images in 10 classes.
  • CIFAR-100: A more complex version with 100 classes.

They tested these datasets across different ResNet architectures (ResNet-20 and ResNet-32) to demonstrate the scalability of their approach to deeper networks.

Key Insights and Ideas

  • Biological Wisdom Meets Engineering: The most important takeaway is that local biological rules (like STDP) don't have to be used in isolation. They can serve as effective "regularizers" or modulators for global gradient-based learning.
  • Efficiency through Sparsity: A recurring challenge in SNNs is "over-spiking," where neurons fire more than necessary, wasting energy. By scaling gradients based on causality, the network learns to be more selective, achieving higher accuracy with fewer total spikes.
  • Compatibility: The proposed Gradient Scaling is a "plug-and-play" addition to existing STBP frameworks, requiring minimal changes to the underlying training architecture.

Key Results

The authors compared their Gradient Scaling (GS) method against a baseline STBP/tdBN approach across various configurations.

  • Accuracy Gains: On CIFAR-10 with ResNet-20 (soft reset), the GS method improved mean accuracy from 94.91% to 95.05%.
  • Spike Reduction: In the same configuration, the number of spikes dropped from 497K to 492K. In some cases, like ResNet-32 with soft reset, spikes were reduced by nearly 4% while maintaining or improving accuracy.
  • SOTA Comparison: As shown in the comparison table, the GS method outperformed several recent SNN training techniques, including calibration and local tandem learning, particularly in terms of achieving high accuracy with very few time steps ($T=4$).
Table 3. Comparisons with previous methods on CIFAR10 (KD: Knowledge Distillation)

Limitations and Future Work

While promising, the study has a few limitations:

  • Simple Scaling Function: The current implementation uses a simple linear interpolation for scaling. The authors suggest that more complex, non-linear scaling functions derived from theoretical analysis might yield even better results.
  • Positive Causality Only: The current model primarily focuses on positive relations between spikes. Incorporating "negative" relations (where a pre-synaptic spike fails to trigger a post-synaptic one) could further refine the learning process, similar to the "Long-Term Depression" (LTD) part of biological STDP.
  • Hyperparameter Sensitivity: The interpolation coefficient $\alpha$ was set empirically to 0.1. Future work could involve automating the tuning of this and other hyperparameters.

Summary updated

Abstract

Deep spiking neural networks (SNNs) are promising neural networks for their model capacity from deep neural network architecture and energy efficiency from SNNs' operations. To train deep SNNs, recently, spatio-temporal backpropagation (STBP) with surrogate gradient was proposed. Although deep SNNs have been successfully trained with STBP, they cannot fully utilize spike information. In this work, we proposed gradient scaling with local spike information, which is the relation between pre- and post-synaptic spikes. Considering the causality between spikes, we could enhance the training performance of deep SNNs. According to our experiments, we could achieve higher accuracy with lower spikes by adopting the gradient scaling on image classification tasks, such as CIFAR10 and CIFAR100.
BibTeX
@misc{park2023gradient,
  author = {Seongsik Park and Jeonghee Jo and Jongkil Park and Yeonjoo Jeong and Jaewook Kim and Suyoun Lee and Joon Young Kwak and Inho Kim and Jong-Keuk Park and Kyeong Seok Lee and Gye Weon Hwang and Hyun Jae Jang},
  title = {{Gradient Scaling on Deep Spiking Neural Networks with Spike-Dependent Local Information}},
  year = {2023},
  eprint = {2308.00558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.00558},
}

Loading related content...

Figure