Ultra-low Latency Adaptive Local Binary Spiking Neural Network with Accuracy Loss Estimator

Changqing Xu, Yijian Pei, Zili Wu, Yi Liu, Yintang YangarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces an Adaptive Local Binary Spiking Neural Network (ALBSNN) that intelligently binarizes network layers to reduce memory and computation while maintaining accuracy. By employing an Accuracy Loss Estimator (ALE) and a Global Average Pooling (GAP) layer, ALBSNN achieves significant storage reduction and ultra-low latency inference without compromising performance on challenging image recognition tasks.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) offer energy efficiency and spatio-temporal processing capabilities, mimicking biological brains. However, their increasing depth leads to memory issues. Binarized SNNs (BSNNs) address this by reducing weights to binary values, but often suffer accuracy loss, typically mitigated by keeping the first and last layers non-binarized. This paper tackles the challenge of finding an optimal binarization strategy that balances accuracy and efficiency, recognizing that a fixed approach (like always keeping the first/last layers non-binarized) might not be optimal for all network architectures and datasets. Furthermore, SNNs traditionally require many time steps for inference, leading to high computational costs, which this paper also aims to reduce.

Methods

The core of this work lies in two novel components:

  • Accuracy Loss Estimator (ALE): This module dynamically determines which layers of the SNN should be binarized. It estimates the error introduced by binarization using the Manhattan distance between original and binarized weights. It also considers the weight storage space and the impact of binarization on both forward and backward passes, using a parabolic function to model this influence. The ALE then uses a selection criterion (based on a calculated R value) to decide whether a layer's weights should be binarized or kept in full precision.
  • Global Average Pooling (GAP) Layer: To address the high latency of SNNs, which often require many time steps for accurate inference, the authors incorporate a GAP layer. This layer, inspired by its success in Convolutional Neural Networks (CNNs), replaces fully connected layers. It adjusts output channels to match the number of classes and then uses global average pooling to convert feature maps into a classification vector, enabling recognition with fewer time steps.

The network itself uses an iterative Leaky Integrate-and-Fire (LIF) neuron model and an approximate gradient for backpropagation.

Datasets

The proposed ALBSNN was evaluated on three standard image recognition datasets:

  • Fashion-MNIST: A dataset of grayscale images of clothing items, with 10 classes.
  • CIFAR-10: A dataset of 3-channel RGB images with 10 classes.
  • CIFAR-100: A dataset of 3-channel RGB images with 100 classes.

These are common benchmarks for evaluating SNNs and are well-established in the field.

Key Insights and Ideas

  • Adaptive Binarization is Crucial: The paper demonstrates that a fixed strategy for binarization (e.g., always keeping the first and last layers non-binarized) is suboptimal. The ALE's ability to adapt binarization based on layer-specific error estimation and network structure is key to maintaining accuracy.
  • Binarized Input as "Beneficial Noise": The authors observe that binarizing the first layer, contrary to common practice, can sometimes improve accuracy. They hypothesize that this binarization acts as a form of "beneficial noise," enhancing the network's robustness.
  • GAP Layer for Latency Reduction: The integration of the GAP layer effectively reduces the number of time steps required for inference, a significant bottleneck for SNNs, without sacrificing accuracy.
  • Trade-offs in Binarization: The paper implicitly highlights the trade-off between the degree of binarization and accuracy. While more binarization saves more resources, it can lead to accuracy drops, which ALE aims to mitigate.

Key Results

  • Accuracy Preservation: ALBSNN achieves competitive accuracy across the tested datasets. For example, on Fashion-MNIST, it achieves 93.10% accuracy, dropping only 0.20% from the full-precision network. On CIFAR-10, it reaches 90.12%, and on CIFAR-100, it achieves 63.54%.
  • Storage Reduction: The method significantly reduces storage space. For CIFAR-10 and CIFAR-100, it achieves more than 20% and 15% reduction, respectively, compared to other methods.
  • Ultra-low Latency: The paper emphasizes achieving high accuracy with a single time step, demonstrating ultra-low latency inference.
  • Improved Performance: Compared to existing methods like Wang-SVGG10, ALBSNN shows an average testing accuracy improvement of 1.44% and 1.52% on CIFAR-10 and CIFAR-100, respectively, with fewer epochs.

Limitations and Future Work

  • Limited Non-Binarized Layers: In the experiments comparing ALBSNN with FLNBSNN, ALBSNN was limited to selecting two layers to maintain full precision. The authors suggest that allowing ALE to select more non-binarized layers could further improve accuracy, especially for complex datasets like CIFAR-100.
  • Selection Criteria Impact: The paper shows that different selection criteria for ALE can lead to varying accuracies. Future work could involve developing more sophisticated criteria or automated methods for selecting the optimal criteria for a given task.
  • Generalizability of ALE: While ALE shows promise, further research could explore its effectiveness on a wider range of network architectures and more complex tasks beyond image recognition.

Summary updated

BibTeX
@misc{xu2022ultra,
  author = {Changqing Xu and Yijian Pei and Zili Wu and Yi Liu and Yintang Yang},
  title = {{Ultra-low Latency Adaptive Local Binary Spiking Neural Network with Accuracy Loss Estimator}},
  year = {2022},
  eprint = {2208.00398},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.00398},
}
Figure