Can we get the best of both Binary Neural Networks and Spiking Neural Networks for Efficient Computer Vision?

Gourav Datta, Zeyu Liu, Peter Anthony Beerel

TL;DR

Researchers have developed a new class of "Sparse Binary Activation Neural Networks" (BANNs) that combine the high energy efficiency of Spiking Neural Networks (SNNs) with the low latency of Binary Neural Networks (BNNs). By using a novel thresholding technique based on the Hoyer regularizer, BANNs achieve state-of-the-art accuracy and extreme activation sparsity (~75%) in just a single time step.

The Problem: Efficiency vs. Sparsity

In the quest for "green AI," two architectures usually dominate the conversation for low-power vision tasks: Binary Neural Networks (BNNs) and Spiking Neural Networks (SNNs).

  • BNNs are highly efficient because they replace expensive floating-point multiplications with cheap "pop-count" operations. However, state-of-the-art BNNs are not actually sparse—every neuron fires every time, meaning computations cannot be skipped to save further energy.
  • SNNs are famous for their activation sparsity. Neurons only fire when they cross a certain threshold, which saves massive amounts of energy on specialized neuromorphic hardware. The catch? To get good accuracy, SNNs usually need to run for multiple "time steps." This ironically increases latency and energy consumption, often negating their theoretical benefits.

This raises an intriguing question: Can we obtain SNN-like sparsity and BNN-like accuracy in a single time step, enjoying the energy-efficiency benefits of both?

Methodology: The Hoyer Threshold

To solve this, the authors introduce BANNs, which use uni-polar binary activations (0 or 1). The secret to their success lies in how they handle the "threshold"—the point at which a neuron decides to fire.

  1. Hoyer Regularizer: Traditionally used for weight pruning, the Hoyer regularizer measures sparsity based on the ratio between the $L_1$ and $L_2$ norms of a tensor. The authors innovatively apply this to activations.
  2. Hoyer Thresholding Layer: Instead of a fixed or purely learned threshold, they set the threshold to the "Hoyer extremum" of a clipped version of the activation map.
  3. Distribution Shifting: By training with this Hoyer-based loss, the network learns to push activation values away from the threshold. This "clears the middle ground," making the binary firing decision much more robust against noise and improving convergence during training.
Hoyer threshold activation and surrogate gradient

Because binary step functions are non-differentiable, standard backpropagation fails. To overcome this, the authors employ a "surrogate gradient" approach—a common trick in SNN literature—to estimate gradients during the training process.

Key Insights

  • Sparsity is a Choice, Not a Burden: The paper proves that you don't need the temporal dimension (multiple time steps) of SNNs to achieve high activation sparsity. You just need a better way to define and train the firing threshold.
  • The "Hoyer Trick": Using the Hoyer extremum as a threshold acts as a natural separator. It encourages the network to produce either very high values (which become 1s) or very low values (which become 0s), leading to high sparsity without sacrificing accuracy.
  • Hardware Portability: While BANNs are designed for single-step efficiency, the method is highly portable. It can be plugged back into multi-step SNNs to boost their performance as well.

Key Results

The framework was rigorously tested across standard benchmarks, including CIFAR-10, ImageNet, object detection tasks (PASCAL VOC), and neuromorphic datasets (DVS-CIFAR10, N-CalTech101).

Spiking activity and energy comparison
  • Accuracy: On CIFAR-10, BANNs achieved 93.44% accuracy with VGG16, outperforming existing one-time-step SNNs and uni-polar BNNs.
  • Sparsity: BANN models consistently yielded a sparsity of ~75% or higher on both CIFAR and ImageNet.
  • Efficiency: Compared to standard Spiking Neural Networks, BANNs achieved a ~5.5× reduction in FLOPs (Floating Point Operations).
  • Energy Savings: On FPGA simulations, the approach demonstrated up to a 70.28× reduction in total energy compared to full-precision models on CIFAR-10, and 81.13× on ImageNet.
  • Object Detection: BANNs outperformed existing BNN and SNN backbones in Mean Average Precision (mAP) for object detection tasks.

Limitations and Future Work

While BANNs beat other uni-polar networks, there is still a small accuracy gap when compared to the absolute best bi-polar BNNs (like ReactNet). Bi-polar networks use -1 and +1, which are more expressive but cannot be sparse. Furthermore, the massive energy gains are most apparent on hardware that can natively "skip" zero-valued activations (like FPGAs or neuromorphic chips); on standard CPUs/GPUs, the benefits may be less dramatic.

Future work will explore more advanced backbone architectures and further optimize the weight-quantization process to close the remaining accuracy gap with full-precision models.

Summary updated

Abstract

Binary Neural networks (BNN) have emerged as an attractive computing paradigm for a wide range of low-power vision tasks. However, state-of-the-art (SOTA) BNNs do not yield any sparsity, and induce a significant number of non-binary operations. On the other hand, activation sparsity can be provided by spiking neural networks (SNN), that too have gained significant traction in recent times. Thanks to this sparsity, SNNs when implemented on neuromorphic hardware, have the potential to be significantly more power-efficient compared to traditional artifical neural networks (ANN). However, SNNs incur multiple time steps to achieve close to SOTA accuracy. Ironically, this increases latency and energy---costs that SNNs were proposed to reduce---and presents itself as a major hurdle in realizing SNNs’ theoretical gains in practice. This raises an intriguing question: *Can we obtain SNN-like sparsity and BNN-like accuracy and enjoy the energy-efficiency benefits of both?* To answer this question, in this paper, we present a training framework for sparse binary activation neural networks (BANN) using a novel variant of the Hoyer regularizer. We estimate the threshold of each BANN layer as the Hoyer extremum of a clipped version of its activation map, where the clipping value is trained using gradient descent with our Hoyer regularizer. This approach shifts the activation values away from the threshold, thereby mitigating the effect of noise that can otherwise degrade the BANN accuracy. Our approach outperforms existing BNNs, SNNs, and adder neural networks (that also avoid energy-expensive multiplication operations similar to BNNs and SNNs) in terms of the accuracy-FLOPs trade-off for complex image recognition tasks. Downstream experiments on object detection further demonstrate the efficacy of our approach. Lastly, we demonstrate the portability of our approach to SNNs with multiple time steps. Codes are publicly available [here](https://github.com/godatta/Ultra-Low-Latency-SNN).
BibTeX
@inproceedings{datta2024can,
  author = {Gourav Datta and Zeyu Liu and Peter Anthony Beerel},
  title = {{Can we get the best of both Binary Neural Networks and Spiking Neural Networks for Efficient Computer Vision?}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=lGUyAuuTYZ},
}
Figure