Neuron Circuits for Low-Power Spiking Neural Networks Using Time-To-First-Spike Encoding

Seongbin Oh, Dongseok Kwon, Gyuho Yeom, Won-Mook Kang, Soochang Lee, Sung Yun Woo, Jaehyeon Kim, Jong-Ho LeeIEEE Access 2022 · DOI

TL;DR

This paper presents a hardware implementation of Spiking Neural Networks (SNNs) using a novel Time-To-First-Spike (TTFS) encoding method. The authors demonstrate that their TTFS-based SNN achieves competitive accuracy on the MNIST dataset while offering significant advantages in power consumption and inference speed compared to traditional rate-encoded SNNs.

What Problem Does This Paper Address?

Artificial Neural Networks (ANNs) have achieved remarkable success, but conventional ANNs are fundamentally different from biological brains, lacking temporal dynamics and being computationally expensive. Spiking Neural Networks (SNNs) offer a more biologically plausible alternative, processing information through discrete spikes and potentially achieving lower power consumption. However, efficiently training and implementing SNNs in hardware, especially with temporal encoding schemes, presents significant challenges. This paper addresses the need for practical, low-power hardware implementations of SNNs that leverage temporal encoding for improved efficiency. Specifically, it focuses on the Time-To-First-Spike (TTFS) encoding method, where information is encoded in the precise timing of the first spike, and explores its hardware realization using novel synaptic devices and neuron circuits.

Methods

The core of this work lies in the hardware implementation and evaluation of an SNN using the Time-To-First-Spike (TTFS) encoding method.

  • Temporal Backpropagation: The authors employ a temporal backpropagation algorithm for training the SNN, which is adapted for the TTFS encoding where information is carried by firing times.
  • TTFS Encoding: Input data is converted into spike trains where the firing time of a neuron is inversely proportional to the input value. Data '0' fires at the last time step.
  • NOR-type Synaptic Device: A NOR-type flash memory device with an asymmetric floating gate is used as the synaptic element. This device offers advantages like resistance to sneak path issues and off-current issues compared to two-terminal devices.
  • Neuron Circuit Design: A novel neuron circuit is proposed, incorporating an integrate-and-fire (I&F) block and a refractory period generator (RPG). The RPG ensures that a neuron fires at most once per input, crucial for TTFS encoding.
  • Circuit Simulation: The entire network, including synaptic devices and neuron circuits, is simulated using SPICE to evaluate its performance under realistic hardware conditions.
  • Hardware Implementation Considerations: The paper also investigates the impact of device variations (device-to-device, threshold variation, stuck-at-off faults) on the network's accuracy, comparing TTFS with rate-encoded networks.

Datasets

  • MNIST: The Modified National Institute of Standards and Technology (MNIST) dataset is used for training and evaluating the performance of the SNN. The images are preprocessed by removing the edges and resizing to 20x20.

Key Insights and Ideas

  • TTFS for Efficiency: The TTFS encoding method inherently leads to sparser network activity compared to rate encoding, which is a key driver for the significant power and latency reductions observed.
  • Hardware-Friendly Synapses: The chosen NOR-type flash memory device is well-suited for synaptic implementation, offering good linearity and resistance to common hardware non-idealities.
  • Refractory Period is Crucial: The inclusion of a refractory period generator in the neuron circuit is essential for correctly implementing the TTFS encoding, ensuring each neuron fires only once per input.
  • Vulnerability to Variations: While TTFS offers efficiency, it can be more sensitive to device variations than rate-encoded networks, especially in networks with fewer hidden neurons, highlighting the need for careful hardware design and tuning.
  • System vs. Circuit Level Alignment: The circuit-level simulations largely validate the system-level simulation results, indicating that the proposed hardware architecture can effectively realize the trained SNN.

Key Results

  • Accuracy: A 2-layer SNN with 512 hidden neurons achieved 96.90% accuracy on the MNIST test set in system-level simulation. In circuit-level simulation, a 128-hidden neuron network achieved 94.9% accuracy, a slight 0.1% drop from system-level simulation.
  • Power Efficiency: The TTFS network consumed 15.12 times lower power than a comparable rate-encoded network when operating with 256 time steps.
  • Latency: The TTFS network could make decisions 5.68 times faster than the rate-encoded network with 256 time steps.
  • Device Variation Impact: The paper quantifies the accuracy degradation due to various device variations, showing that TTFS networks can be more susceptible than rate-encoded ones, especially with fewer hidden neurons.

Limitations and Future Work

  • Device Variation Sensitivity: The paper acknowledges that TTFS networks can be more sensitive to device variations than rate-encoded networks, particularly when the number of hidden neurons is small. Future work could focus on developing more robust training methods or hardware designs to mitigate these effects.
  • I&F Block Leakage: The authors note that the I&F block contributes significantly to standby power due to subthreshold leakage. Improving the I&F circuit design to reduce this leakage is suggested as a future research direction.
  • Off-Current Neglect: The system-level simulation did not account for off-current in the synaptic array, which is cited as a reason for the slight accuracy drop in circuit-level simulations. Future work could incorporate this into system-level models for better prediction.
  • Further Exploration of Variations: While several variations were analyzed, a deeper investigation into their combined effects and mitigation strategies could be beneficial.

Summary updated

BibTeX
@article{oh2022neuron,
  author = {Seongbin Oh and Dongseok Kwon and Gyuho Yeom and Won-Mook Kang and Soochang Lee and Sung Yun Woo and Jaehyeon Kim and Jong-Ho Lee},
  title = {{Neuron Circuits for Low-Power Spiking Neural Networks Using Time-To-First-Spike Encoding}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3149577},
}

Loading related content...

Figure