Spiking Neural Networks (SNNs) promise exceptional energy efficiency by processing information via sparse, discrete events (spikes) rather than continuous activations. However, as noted by Jason Eshraghian et al., SNNs historically struggle to capture long-term associations in sequential data compared to modern artificial neural networks (ANNs) like Transformers or LSTMs.
This fundamental limitation stems from a severe optimization bottleneck: temporal gradient vanishing. When training SNNs using Backpropagation Through Time (BPTT), the mathematical formulation of standard spiking neurons causes error signals to decay exponentially over time, preventing the network from linking distant causes and effects.
The Root Cause: Leaky Integration and Hard Resets
To understand why gradients vanish, we must look at the forward dynamics of a standard Leaky Integrate-and-Fire (LIF) neuron. The membrane potential $H_t$ at time step $t$ is governed by a decay constant $\beta \in (0, 1)$—reflecting the leakage of charge over time—and incoming synaptic inputs. When $H_t$ crosses a threshold, the neuron emits a spike and its potential is reset.
During BPTT, the gradient of the loss with respect to an earlier state $H_{t-k}$ relies on the chain rule product:
$$\frac{\partial L}{\partial H_{t-k}} = \frac{\partial L}{\partial H_t} \prod_{i=1}^k \frac{\partial H_{t-i+1}}{\partial H_{t-i}}$$
As analyzed by Yulong Huang et al., the leaky integration mechanism introduces this decay constant $\beta$ as a direct multiplier at every single temporal step. Because $\beta < 1$, the gradient signal attenuates exponentially as it travels backward through time.
Compounding this issue, the standard "hard reset" mechanism abruptly clears the membrane potential whenever a spike occurs. As highlighted by Shuaijie Shen et al. and Lang Qin et al., these discrete resets interrupt the continuous accumulation of information, purging the historical context required to maintain long-range dependencies. Consequently, networks suffer from vanishing signals where temporal activity and credit assignment diminish over extended horizons, making it difficult to optimize parameters for early inputs in a sequence (Jiahao Su et al.).
Architectural Remedies: Compartmentalization and Gating
To prevent information from dissipating, researchers have redesigned the internal architecture of spiking neurons to isolate memory retention from spike generation.
One prominent approach mimics biological neural structures. Shimin Zhang et al. (2023) and Shimin Zhang et al. (2024) introduced the Two-Compartment Leaky Integrate-and-Fire (TC-LIF) model. By separating the neuron into a dendritic compartment (which handles long-term context integration without immediate decay) and a somatic compartment (which handles spike generation), the model prevents the gradient from vanishing, allowing the network to successfully perform long-term temporal credit assignment.
Standard LIF:
Input ---> [ Leaky Integration & Reset ] ---> Spike (Gradients vanish via decay & reset)
TC-LIF:
Input ---> [ Dendritic Compartment ] ---------> [ Somatic Compartment ] ---> Spike
(Preserves Context) (Generates Spikes)
Alternatively, researchers have integrated gating mechanisms directly into the spiking dynamics, drawing inspiration from LSTMs. Kaushik Roy et al. and Lang Qin et al. developed gated recurrent spiking neurons (such as GRSN) that dynamically regulate information flow. By utilizing learnable gates to control how much historical state is retained or reset, these models create a constant error pathway within the spiking framework, ensuring that gradients can flow back across long sequences without exponential decay. Similarly, Fenghao Liu et al. proposed the LIF Recurrent Memory Module (LRMM), a specialized micro-circuit of simple leaky neurons that acts as a robust memory buffer over long horizons.
Mathematical Reformulations: Complementary Potentials and Parallelization
Rather than adding complex gates, another strategy modifies the underlying state equations to preserve the gradient mathematically.
Yulong Huang et al. developed the Complementary Leaky Integrate-and-Fire (CLIF) neuron. CLIF introduces a secondary, hyperparameter-free "complementary potential" that explicitly tracks the exact amount of information lost to decay at each step. During backpropagation, this complementary path acts as a counter-weight, capturing the attenuated gradient and allowing error signals to remain stable over long time scales.
A more radical departure bypasses sequential BPTT entirely by eliminating the serial dependency chain. Wei Fang et al. introduced the Parallel Spiking Neuron (PSN), which removes the traditional sequential reset mechanism and reformulates the neuronal dynamics into parallelizable matrix operations. This prevents the gradient from exploding or vanishing across long chains of serial updates, mirroring the training efficiencies found in modern non-recurrent architectures.
Bridging SNNs with State Space Models
The most recent paradigm shift involves merging SNNs with State Space Models (SSMs), which are inherently designed to handle long-range dependencies across thousands of steps.
Shuaijie Shen et al. proposed SpikingSSMs, which leverage a surrogate dynamic network to parallelize training and maintain continuous information accumulation, avoiding the disruptive resets of standard SNNs. Concurrently, Malyaban Bal et al. developed P-SpikeSSM, which pairs a parallelizable, probabilistic spike-sampling mechanism with a state-space formulation. Because SSMs possess a linear state transition that can be computed efficiently in parallel, these architectures completely circumvent the temporal gradient bottleneck of BPTT while preserving the sparse, energy-efficient execution of neuromorphic hardware.
Alternative Optimization and Structural Paradigms
Beyond changing the single-neuron equations, researchers are exploring holistic training and structural modifications to stabilize temporal learning:
- Neural Oscillations: Yinsong Yan et al. introduced Rhythm-SNN, which utilizes brain-inspired neural oscillations to modulate spiking activity, stabilizing gradient signals over extended sequences.
- Brain Topology: From a structural perspective, Yongjian Wang et al. demonstrated that incorporating real biological brain topologies can improve reinforcement learning performance and mitigate gradient issues in recurrent architectures.
- Loss Landscape Smoothing: To address the optimization landscape directly, Desong Zhang et al. introduced Dominant Singular Deflation (DSD) to prevent gradient alignment and sharp loss landscapes that often plague direct-encoded SNNs over time.
By rethinking the interplay between leaky integration, reset mechanisms, and backpropagation, these diverse approaches ensure that temporal credit assignment can be computed reliably over long horizons, unlocking the full potential of SNNs for complex sequential tasks.
Key Foundational Work
TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling (2023) — contributed 1 concept elements to this direction.
Parallel Spiking Neurons with High Efficiency and Ability to Learn Long-term Dependencies (2023) — contributed 1 concept elements to this direction.
Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning (2022) — contributed 1 concept elements to this direction.