A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks

Jibin Wu, Yansong Chua, Malu Zhang, Guoqi Li, Haizhou Li, Kay Chen TanIEEE TNNLS 2023 · DOI · OpenReview

TL;DR

This paper introduces a novel "tandem learning" framework that pairs a Spiking Neural Network (SNN) with a standard Artificial Neural Network (ANN). This allows for efficient training of SNNs using backpropagation while maintaining the SNN's inherent advantages for rapid, low-power inference on neuromorphic hardware. The approach achieves competitive accuracy on various vision tasks with significantly reduced inference times and computational costs.

What Problem Does This Paper Address?

Deep learning has revolutionized computer vision, but traditional Artificial Neural Networks (ANNs) are computationally intensive and power-hungry, limiting their deployment on resource-constrained devices. Spiking Neural Networks (SNNs), inspired by the brain's efficient, event-driven processing, offer a promising alternative for neuromorphic computing. However, the non-differentiable nature of spiking neurons makes standard backpropagation difficult, hindering their training. Existing methods like ANN-to-SNN conversion or constrain-then-train approaches often require long inference times or suffer accuracy drops. This paper aims to bridge this gap by enabling effective and efficient training of deep SNNs for rapid, low-power inference.

Methods

The core innovation is the tandem learning framework. It consists of two networks trained in parallel:

  • Spiking Neural Network (SNN): This network handles the actual spike generation and determines the discrete neural representations (spike counts and spike trains).
  • Artificial Neural Network (ANN): This network acts as an auxiliary structure. It shares weights with the SNN and is designed with custom activation functions that approximate the SNN's spike counts. This ANN allows for standard error backpropagation.

During training:

  1. Forward Pass: The SNN generates spike counts and spike trains. These are fed into both the SNN and the coupled ANN.
  2. Backward Pass: Error gradients are computed and backpropagated through the ANN. These gradients are then used to update the shared weights, which are subsequently applied to the SNN.

This "tandem" approach allows the SNN to learn accurate spike-based representations while leveraging the ANN for efficient gradient computation, effectively enabling spike-train level learning. The paper also explores different neuron models (Integrate-and-Fire (IF) and Leaky Integrate-and-Fire (LIF)) and their corresponding ANN approximations.

Datasets

The authors evaluated their method on several standard datasets:

  • Frame-based:
    • MNIST (handwritten digits)
    • CIFAR-10 (object recognition)
    • ImageNet-12 (large-scale object recognition)
  • Event-based:
    • N-MNIST (event-based MNIST)
    • DVS-CIFAR10 (event-based CIFAR-10)

Key Insights and Ideas

  • Bridging the SNN Training Gap: The tandem learning approach effectively circumvents the non-differentiability issue of SNNs by using a coupled ANN for backpropagation, enabling spike-train level learning.
  • Spike Count as Discrete Representation: The paper highlights the effectiveness of using spike counts as a discrete latent representation in SNNs, which can be approximated by ANN activation functions.
  • Efficiency without Sacrificing Accuracy: The method achieves competitive classification accuracy while drastically reducing inference time and computational cost compared to other SNN training methods.
  • Temporal Dynamics vs. Spike Count: While the method primarily relies on spike counts, it still performs well, suggesting that for many tasks, precise temporal dynamics might not be strictly necessary for achieving high accuracy, especially when combined with efficient encoding.
  • Scalability: The tandem learning rule is shown to be scalable to large datasets like ImageNet-12, a significant challenge for many SNN training methods.

Key Results

  • Object Recognition: Achieved competitive accuracies on CIFAR-10 (up to 90.98% with IF neurons and aggregate membrane potential decoding) and N-MNIST (99.31% with IF neurons).
  • ImageNet-12: Achieved top-1 accuracy of 46.63% (70.80% with ANN approximation) with only 5 inference time steps, significantly faster than other methods.
  • Inference Speed: Demonstrated at least an order of magnitude reduction in inference time compared to other SNN implementations.
  • Computational Efficiency: Achieved significantly lower total synaptic operations (SynOps) compared to ANNs and other SNN methods, leading to potential for lower power consumption.
  • Regression: Showed strong performance on image reconstruction tasks on MNIST, achieving high-quality reconstructions with minimal MSE drop compared to ANNs.

Limitations and Future Work

  • Temporal Sequence Learning: The current tandem learning rule, which focuses on spike counts, is not directly applicable to tasks requiring precise temporal sequence learning (e.g., gesture recognition). The authors suggest future work could involve hybrid architectures combining feedforward SNNs with recurrent SNNs for such tasks.
  • LIF Neuron Approximation: While the method works well, the approximation for LIF neurons might still have room for improvement. Future work could involve designing more effective approximating functions for LIF neurons.
  • Network Architectures: The authors suggest evaluating more advanced network architectures with their tandem learning rule.
  • Data Scarcity for Event-Based Vision: For event-based datasets, the authors explored transfer learning due to data scarcity, indicating a potential area for further research.

Summary updated

Abstract

Spiking neural networks (SNNs) represent the most prominent biologically inspired computing model for neuromorphic computing (NC) architectures. However, due to the nondifferentiable nature of spiking neuronal functions, the standard error backpropagation algorithm is not directly applicable to SNNs. In this work, we propose a tandem learning framework that consists of an SNN and an artificial neural network (ANN) coupled through weight sharing. The ANN is an auxiliary structure that facilitates the error backpropagation for the training of the SNN at the spike-train level. To this end, we consider the spike count as the discrete neural representation in the SNN and design an ANN neuronal activation function that can effectively approximate the spike count of the coupled SNN. The proposed tandem learning rule demonstrates competitive pattern recognition and regression capabilities on both the conventional frame- and event-based vision datasets, with at least an order of magnitude reduced inference time and total synaptic operations over other state-of-the-art SNN implementations. Therefore, the proposed tandem learning rule offers a novel solution to training efficient, low latency, and high-accuracy deep SNNs with low computing resources.
BibTeX
@article{wu2023tandem,
  author = {Jibin Wu and Yansong Chua and Malu Zhang and Guoqi Li and Haizhou Li and Kay Chen Tan},
  title = {{A Tandem Learning Rule for Effective Training and Rapid Inference of Deep Spiking Neural Networks}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2023},
  doi = {10.1109/tnnls.2021.3095724},
}
Figure