ASRC-SNN: Adaptive Skip Recurrent Connection Spiking Neural Network

Shang Xu, Jiayu Zhang, Ziming Wang, Runhao Jiang, Rui Yan, Huajin TangSubmission to ICML 2025 · Rejected

TL;DR

Vanilla Recurrent Spiking Neural Networks (RSNNs) struggle with long-term temporal dependencies due to severe gradient vanishing across time steps. By recognizing that leaky integrate-and-fire (LIF) neurons already handle adjacent-step connections through their intrinsic decay dynamics, this paper introduces Adaptive Skip Recurrent Connections (ASRC-SNN), which adaptively learn layer-specific long-range temporal skip spans using temperature-annealed Softmax kernels. The resulting models establish state-of-the-art accuracy on sequential and speech SNN benchmarks with significantly fewer parameters.

What Problem Does This Paper Address?

While Spiking Neural Networks (SNNs) excel at energy-efficient neuromorphic computation, they typically underperform traditional artificial neural networks (ANNs) on complex sequence tasks like speech and long temporal modeling. Prior works attempted to fix this by designing intricate, multi-compartment neuron models while simply attaching a standard Elman-style recurrent connection ($t-1$) as an afterthought.

The authors demonstrate that viewing the neuron and recurrence as an integrated dynamical system exposes a fundamental flaw: gradients propagating backward through time vanish exponentially due to the multiplicative decay of both the LIF membrane potential and the recurrent weights.

Methods

  • Gradient Propagation Analysis: The authors formally analyze temporal backpropagation in LIF-based RSNNs. They show that while orthogonal initialization mitigates gradient explosion, avoiding gradient vanishing requires impossibly strict conditions ($|W_2^l| = V_{th}$ and saturated surrogate gradients across time).
  • Skip Recurrent Connections (SRC): Instead of connecting time step $t$ to $t-1$, SRC connects step $t$ directly to $t-\lambda$ (where $\lambda > 1$). Crucially, adjacent-step continuity ($t-1$) is not lost—it is already naturally maintained by the LIF neuron's intrinsic membrane potential decay.
  • Adaptive Skip Recurrent Connections (ASRC): Manually tuning a fixed $\lambda$ per layer is intractable and dataset-dependent. ASRC assigns a vector of learnable weights over a historical window of length $T_\lambda$. A temperature-scaled Softmax kernel distributes connection weights: $$\text{Softmax}_\tau(w_i) = \frac{\exp(w_i / \tau)}{\sum_j \exp(w_j / \tau)}$$ As training progresses, the temperature $\tau$ exponentially decays (factor of $0.96$ per epoch). Initially, the smooth distribution explores multiple temporal lags; by test time, it converges toward a discrete Hardmax (one-hot) skip connection customized for each layer.

Datasets

The authors evaluated the method across four standard temporal classification benchmarks:

  1. Sequential MNIST (S-MNIST): Pixels fed as a sequence of length 784 ($28 \times 28$).
  2. Permuted Sequential MNIST (PS-MNIST): S-MNIST with a fixed random sequence permutation, destroying local spatial correlations to test long-range temporal retention.
  3. Spiking Google Speech Commands (SSC): A neuromorphic audio dataset with 35 classes, 140 input channels, and 250 time steps.
  4. Google Speech Commands v0.01 (GSC): Audio converted to Mel-spectrograms (decibel scale) for temporal keyword spotting.

No new datasets were introduced.

Key Insights and Ideas

  • Recurrence and Neuron Synergy: Recurrent connections in SNNs do not need to connect immediate neighbors ($t-1$). Because LIF neurons already possess "self-recurrent" memory through membrane leakage, recurrent weights are better spent bridging wider temporal gaps.
  • Continuous Relaxation to Discrete Architectures: Learning discrete temporal delays is notoriously difficult with gradient descent. Annealing a temperature-scaled Softmax provides an effective continuous relaxation that smoothly hardens into a single integer delay per layer without discrete optimization tricks.
  • Robustness Across Scales: Unlike fixed-skip networks (SRC-SNN)—which are brittle and sensitive to the chosen value of $\lambda$—ASRC-SNN performance stabilizes once the maximum window $T_\lambda$ is sufficiently large, automatically finding optimal layer-wise skip patterns.

Key Results

ASRC-SNN achieved state-of-the-art results among recurrent SNN architectures while utilizing compact three-layer fully connected networks:

  • S-MNIST: Reached 99.57% accuracy (0.063M parameters), outperforming prior multi-compartment neuron models like TC-LIF (99.20%) and PMSN (99.53%).
  • PS-MNIST: Achieved 96.62% (0.15M parameters), surpassing all previous recurrent SNN approaches (GLIF at 90.47%, ALIF at 94.30%, TC-LIF at 95.36%).
  • SSC: Outperformed specialized delay-learning and recurrent architectures, scoring 81.91% accuracy (compared to SpikGRU's 77.00% and DCLS-Delays' 80.69%).
  • GSC: Achieved 96.29% with only 0.089M parameters, outperforming TC-LIF (94.84% with 0.19M) and SNN with SFA (91.21% with 4.3M).

Limitations and Future Work

  • Single Discrete Connection: In the current formulation, the temperature cooling forces the kernel to collapse onto a single discrete delay per layer, potentially discarding multi-scale historical information observed during intermediate training phases.
  • Future Directions:
  • Extending ASRC to learn discrete positions across both time and space (e.g., learnable non-zero offsets in dilated convolutions).
  • Removing the temperature annealing schedule by penalizing the variance of the Softmax distribution directly in the loss function.
  • Developing adaptive multi-skip recurrent connections that allow neurons to maintain multiple learned historical connection points simultaneously.

Summary updated

Abstract

In recent years, Recurrent Spiking Neural Networks (RSNNs) have shown promising potential in long-term temporal modeling. Many studies focus on improving neuron models and also integrate recurrent structures, leveraging their synergistic effects to improve the long-term temporal modeling capabilities of Spiking Neural Networks (SNNs). However, these studies often place an excessive emphasis on the role of neurons, overlooking the importance of analyzing neurons and recurrent structures as an integrated framework. In this work, we consider neurons and recurrent structures as an integrated system and conduct a systematic analysis of gradient propagation along the temporal dimension, revealing a challenging gradient vanishing problem. To address this issue, we propose the Skip Recurrent Connection (SRC) as a replacement for the vanilla recurrent structure, effectively mitigating gradient vanishing problem and enhancing long-term temporal modeling performance. Additionally, we propose the Adaptive Skip Recurrent Connection (ASRC), a method that can learn the skip span of skip recurrent connection in each layer of the network. Experiments show that replacing the vanilla recurrent structure in RSNN with SRC significantly improves the model's performance on temporal benchmark datasets. Moreover, ASRC-SNN outperforms SRC-SNN in terms of temporal modeling capabilities and robustness.
BibTeX
@misc{xu2025asrc,
  author = {Shang Xu and Jiayu Zhang and Ziming Wang and Runhao Jiang and Rui Yan and Huajin Tang},
  title = {{ASRC-SNN: Adaptive Skip Recurrent Connection Spiking Neural Network}},
  year = {2025},
  eprint = {2505.11455},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.11455},
}
Figure