Binary stochasticity enabled highly efficient neuromorphic deep learning achieves better-than-software accuracy

Yang Li, Wei Wang, Ming Wang, Chunmeng Dou, Zhengyu Ma, Huihui Zhou, Peng Zhang, Nicola Lepri, Xumeng Zhang, Qing Luo, Xiaoxin Xu, Guanhua Yang, Feng Zhang, Ling Li, Daniele Ielmini, Ming LiuAdv. Intell. Syst 2023 · DOI · OpenReview

TL;DR

This paper introduces a novel "binary stochastic learning" algorithm that significantly simplifies neural network operations for neuromorphic hardware. By binarizing signals and derivatives and using only the sign of errors, they achieve comparable or better accuracy than traditional high-precision methods while drastically reducing energy consumption and hardware complexity, paving the way for more efficient and biologically plausible AI.

What Problem Does This Paper Address?

Deep learning, while powerful, demands high precision in computations for signal propagation, error backpropagation, and weight updates. This high precision is a major bottleneck for implementing deep learning on neuromorphic hardware, especially those using analog components like memristors. These analog components are inherently noisy and prone to variations, making precise analog-to-digital and digital-to-analog conversions computationally expensive and power-hungry. The authors aim to break this dependency on high precision, demonstrating that a more "noisy" and simplified approach can lead to more efficient and even better-performing deep learning systems, particularly for hardware implementations.

Methods

The core innovation is the "binary stochastic learning" algorithm, which modifies standard deep learning operations:

  • Stochastic Binarization of Forwarding Signals: Instead of transmitting precise values, signals are binarized (0 or 1) based on a probability derived from the neuron's activation. This is implemented using a Bernoulli process.
  • Stochastic Binarization of Activation Derivatives: The derivatives of activation functions, crucial for backpropagation, are also stochastically binarized.
  • Signed Binarization of Backpropagating Errors: Only the sign (positive or negative) of the error signals is backpropagated, discarding magnitude information.
  • Step-wise Weight Updates: A periodical carry method is used for weight updates, which is compatible with quantized or noisy analog weights and simplifies the process.

These techniques are designed to be hardware-friendly, reducing the need for complex peripheral circuits and high-precision converters.

Datasets

  • MNIST: Used for training and testing fully-connected and convolutional neural networks.
  • CIFAR-10: Used for training and testing deeper convolutional neural networks.

Key Insights and Ideas

  • High Precision is Not Always Necessary: The paper challenges the assumption that high precision is a prerequisite for effective deep learning, showing that a stochastic, simplified approach can yield superior or comparable results.
  • Hardware Friendliness: The proposed binary stochastic methods are explicitly designed to leverage the characteristics of neuromorphic hardware (like memristors), minimizing the need for power-hungry digital-to-analog and analog-to-digital converters.
  • Energy Efficiency Breakthrough: By simplifying operations, the energy consumption for multiply-accumulate (MAC) operations can be reduced by orders of magnitude, making large-scale neuromorphic deployments more feasible.
  • Improved Generalization: Binary stochastic learning demonstrated better generalization capabilities on unseen data compared to high-precision learning, suggesting it might be less prone to overfitting.
  • Biological Plausibility: The stochastic nature of the proposed approach aligns better with the inherent noise and probabilistic firing of biological neurons.

Key Results

  • Accuracy: The binary stochastic learning algorithm achieved comparable or better accuracy than high-precision learning on MNIST and CIFAR-10 datasets. In some cases, it even surpassed software-based benchmarks.
  • Energy Efficiency:
    • CMOS implementation: MAC operations reduced from 4.6 pJ to 0.9 pJ (with FP32 weights). Quantized weights further reduced this to as low as 5.6 fJ.
  • Memristor-based neuromorphic system: Energy consumption for MAC operations reduced by 100x (from 0.18 pJ to 1.8 fJ) and overall energy reduction by more than three orders of magnitude compared to conventional memristive systems.
  • Footprint Reduction: Significant reduction in chip area was projected, by up to 57.8 times.
  • Inference Performance: Stochastic inference, especially with repeated sampling and majority voting, showed improved accuracy and reduced error rates, sometimes outperforming high-precision inference.
  • Convolutional Networks: The binary stochastic learning approach was effective for deep convolutional neural networks, showing better learning performance and avoiding overfitting.

Limitations and Future Work

  • Neural Network Depth: The paper notes that binary stochastic learning becomes difficult for very deep neural networks (more than 10 layers) due to information loss through long chains of stochastic binarization.
  • Future Work: The authors suggest further investigation into extending the algorithms to deeper networks. They also propose pre-training in high-precision format or increasing batch size as potential compensation strategies for deeper networks. The biological plausibility of the approach is highlighted as a key area for future exploration.

Summary updated

Abstract

Deep learning needs high-precision handling of forwarding signals, backpropagating errors, and updating weights. This is inherently required by the learning algorithm since the gradient descent learning rule relies on the chain product of partial derivatives. However, it is challenging to implement deep learning in hardware systems that use noisy analog memristors as artificial synapses, as well as not being biologically plausible. Memristor-based implementations generally result in an excessive cost of neuronal circuits and stringent demands for idealized synaptic devices. Here, we demonstrate that the requirement for high precision is not necessary and that more efficient deep learning can be achieved when this requirement is lifted. We propose a binary stochastic learning algorithm that modifies all elementary neural network operations, by introducing (i) stochastic binarization of both the forwarding signals and the activation function derivatives, (ii) signed binarization of the backpropagating errors, and (iii) step-wised weight updates. Through an extensive hybrid approach of software simulation and hardware experiments, we find that binary stochastic deep learning systems can provide better performance than the software-based benchmarks using the high-precision learning algorithm. Also, the binary stochastic algorithm strongly simplifies the neural network operations in hardware, resulting in an improvement of the energy efficiency for the multiply-and-accumulate operations by more than three orders of magnitudes.
BibTeX
@article{li2023binary,
  author = {Yang Li and Wei Wang and Ming Wang and Chunmeng Dou and Zhengyu Ma and Huihui Zhou and Peng Zhang and Nicola Lepri and Xumeng Zhang and Qing Luo and Xiaoxin Xu and Guanhua Yang and Feng Zhang and Ling Li and Daniele Ielmini and Ming Liu},
  title = {{Binary stochasticity enabled highly efficient neuromorphic deep learning achieves better-than-software accuracy}},
  journal = {Adv. Intell. Syst.},
  year = {2023},
  doi = {10.1002/aisy.202300399},
}
Figure