Abstract
Spiking Neural Networks (SNNs) offer a promising path toward energy-efficient neuromorphic computing but often struggle with sequential learning tasks due to vanishing gradients and non-differentiable activation functions. This paper introduces improved inherent recurrence dynamics by incorporating "forget gates" into SNN synaptic updates and utilizing multi-bit spiking to mitigate gradient mismatch. Evaluated on speech recognition benchmarks like TIMIT and LibriSpeech, the proposed models achieve performance comparable to LSTMs while requiring 2x fewer parameters and over 10x fewer multiplication operations than GRUs.
Introduction
Neuromorphic computing leverages Spiking Neural Networks (SNNs) to mimic the event-driven, sparse communication of the biological brain. While SNNs promise massive power savings on specialized hardware, their application to complex sequential learning—such as speech recognition—has been limited. The primary challenge lies in the "leaky" nature of standard Leaky Integrate-and-Fire (LIF) neurons, where internal states decay too quickly, leading to the vanishing gradient problem. Additionally, the discrete, non-differentiable nature of spikes complicates training via standard backpropagation.
Methods: Enhancing SNN Dynamics
To bridge the gap between SNNs and traditional Recurrent Neural Networks (RNNs), this work proposes two key architectural advancements focused on memory retention and gradient stability.
1. Improved Inherent Recurrence
The authors argue that SNNs are naturally recurrent systems where membrane potential and synaptic currents act as hidden states. By modifying the synaptic current equations to include a forget gate and a candidate signal, the network gains the ability to selectively retain or discard information over long sequences.
In the "v2" iteration of this model, these gates are driven by the sparse spiking output ($Y[n-1]$) of the previous time step rather than dense internal states. This ensures the network remains event-driven while gaining the performance benefits of gated architectures like LSTMs or GRUs.
2. Multi-bit Spiking and Surrogate Gradients
Training SNNs requires handling the non-differentiable step function of a spike. This is typically managed using surrogate gradients during the backward pass to approximate the derivative.
To further improve training stability, the authors transition from binary spikes to multi-bit outputs (e.g., 6-bit). This significantly reduces "gradient mismatch"—the error introduced when a smooth surrogate derivative tries to model a sharp step-function. A dynamic thresholding scheme using an Exponential Moving Average (EMA) of membrane potential statistics is also employed to ensure robustness across different datasets without manual tuning.
Experimental Results
The proposed gated SNNs were evaluated on the TIMIT (phoneme recognition) and LibriSpeech 100h (word recognition) datasets to test their ability to handle real-world temporal sequences.
- Accuracy: The SNN (v2) achieved accuracy within 1.10% of LSTMs on TIMIT and within 0.36% on LibriSpeech.
- Parameter Efficiency: The model required 2x fewer parameters than LSTMs to achieve these results.
- Computational Savings: Due to high output sparsity (roughly 84-86% zeros), the SNN achieved over 10x savings in multiplication operations compared to GRUs (10.13x on TIMIT and 11.14x on LibriSpeech).
Conclusion
By treating SNNs as inherent RNNs and managing their internal states with gates, this research unlocks their potential for complex sequential tasks. The combination of gated dynamics and multi-bit spiking provides a high-performance, energy-efficient alternative to traditional RNNs, particularly suited for real-time edge processing where power and parameter constraints are paramount.
Summary updated