TL;DR
Vanilla Recurrent Spiking Neural Networks (RSNNs) struggle with long-term temporal dependencies due to severe gradient vanishing across time steps. By recognizing that leaky integrate-and-fire (LIF) neurons already handle adjacent-step connections through their intrinsic decay dynamics, this paper introduces Adaptive Skip Recurrent Connections (ASRC-SNN), which adaptively learn layer-specific long-range temporal skip spans using temperature-annealed Softmax kernels. The resulting models establish state-of-the-art accuracy on sequential and speech SNN benchmarks with significantly fewer parameters.
What Problem Does This Paper Address?
While Spiking Neural Networks (SNNs) excel at energy-efficient neuromorphic computation, they typically underperform traditional artificial neural networks (ANNs) on complex sequence tasks like speech and long temporal modeling. Prior works attempted to fix this by designing intricate, multi-compartment neuron models while simply attaching a standard Elman-style recurrent connection ($t-1$) as an afterthought.
The authors demonstrate that viewing the neuron and recurrence as an integrated dynamical system exposes a fundamental flaw: gradients propagating backward through time vanish exponentially due to the multiplicative decay of both the LIF membrane potential and the recurrent weights.
Methods
- Gradient Propagation Analysis: The authors formally analyze temporal backpropagation in LIF-based RSNNs. They show that while orthogonal initialization mitigates gradient explosion, avoiding gradient vanishing requires impossibly strict conditions ($|W_2^l| = V_{th}$ and saturated surrogate gradients across time).
- Skip Recurrent Connections (SRC): Instead of connecting time step $t$ to $t-1$, SRC connects step $t$ directly to $t-\lambda$ (where $\lambda > 1$). Crucially, adjacent-step continuity ($t-1$) is not lost—it is already naturally maintained by the LIF neuron's intrinsic membrane potential decay.
- Adaptive Skip Recurrent Connections (ASRC): Manually tuning a fixed $\lambda$ per layer is intractable and dataset-dependent. ASRC assigns a vector of learnable weights over a historical window of length $T_\lambda$. A temperature-scaled Softmax kernel distributes connection weights: $$\text{Softmax}_\tau(w_i) = \frac{\exp(w_i / \tau)}{\sum_j \exp(w_j / \tau)}$$ As training progresses, the temperature $\tau$ exponentially decays (factor of $0.96$ per epoch). Initially, the smooth distribution explores multiple temporal lags; by test time, it converges toward a discrete Hardmax (one-hot) skip connection customized for each layer.
Datasets
The authors evaluated the method across four standard temporal classification benchmarks:
- Sequential MNIST (S-MNIST): Pixels fed as a sequence of length 784 ($28 \times 28$).
- Permuted Sequential MNIST (PS-MNIST): S-MNIST with a fixed random sequence permutation, destroying local spatial correlations to test long-range temporal retention.
- Spiking Google Speech Commands (SSC): A neuromorphic audio dataset with 35 classes, 140 input channels, and 250 time steps.
- Google Speech Commands v0.01 (GSC): Audio converted to Mel-spectrograms (decibel scale) for temporal keyword spotting.
No new datasets were introduced.
Key Insights and Ideas
- Recurrence and Neuron Synergy: Recurrent connections in SNNs do not need to connect immediate neighbors ($t-1$). Because LIF neurons already possess "self-recurrent" memory through membrane leakage, recurrent weights are better spent bridging wider temporal gaps.
- Continuous Relaxation to Discrete Architectures: Learning discrete temporal delays is notoriously difficult with gradient descent. Annealing a temperature-scaled Softmax provides an effective continuous relaxation that smoothly hardens into a single integer delay per layer without discrete optimization tricks.
- Robustness Across Scales: Unlike fixed-skip networks (SRC-SNN)—which are brittle and sensitive to the chosen value of $\lambda$—ASRC-SNN performance stabilizes once the maximum window $T_\lambda$ is sufficiently large, automatically finding optimal layer-wise skip patterns.
Key Results
ASRC-SNN achieved state-of-the-art results among recurrent SNN architectures while utilizing compact three-layer fully connected networks:
- S-MNIST: Reached 99.57% accuracy (0.063M parameters), outperforming prior multi-compartment neuron models like TC-LIF (99.20%) and PMSN (99.53%).
- PS-MNIST: Achieved 96.62% (0.15M parameters), surpassing all previous recurrent SNN approaches (GLIF at 90.47%, ALIF at 94.30%, TC-LIF at 95.36%).
- SSC: Outperformed specialized delay-learning and recurrent architectures, scoring 81.91% accuracy (compared to SpikGRU's 77.00% and DCLS-Delays' 80.69%).
- GSC: Achieved 96.29% with only 0.089M parameters, outperforming TC-LIF (94.84% with 0.19M) and SNN with SFA (91.21% with 4.3M).
Limitations and Future Work
- Single Discrete Connection: In the current formulation, the temperature cooling forces the kernel to collapse onto a single discrete delay per layer, potentially discarding multi-scale historical information observed during intermediate training phases.
- Future Directions:
- Extending ASRC to learn discrete positions across both time and space (e.g., learnable non-zero offsets in dilated convolutions).
- Removing the temperature annealing schedule by penalizing the variance of the Softmax distribution directly in the loss function.
- Developing adaptive multi-skip recurrent connections that allow neurons to maintain multiple learned historical connection points simultaneously.
Summary updated