Spiking Neural Networks (SNNs) offer a highly energy-efficient alternative to traditional Artificial Neural Networks (ANNs) by mimicking the brain's event-driven communication. Instead of processing continuous activation values, SNN neurons communicate via discrete, binary pulses (spikes) over time.
However, scaling SNNs to the depths commonly used in modern computer vision and sequence modeling introduces a severe bottleneck. As information propagates through successive layers, neural activity frequently dissipates, and gradients drop to zero. This dual failure of the forward and backward passes is known as the vanishing spike problem or spike degradation. Understanding why this happens—and how recent research resolves it—is the key to unlocking deep, high-performance neuromorphic architectures.
The Forward Pass: Why Spikes Vanish
In a standard ANN, continuous activations pass through layers without losing their fundamental resolution. In an SNN, however, a neuron only fires a spike ($1$) if its internal membrane potential crosses a strict threshold; otherwise, it outputs a $0$.
This binary quantization creates a severe information bottleneck. As signals travel deeper into the network, the variance of neural activity tends to dissipate, and the mean firing rate often drops toward zero. As demonstrated by Yi Zeng et al. in Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization, this progressive loss of signal prevents deeper layers from maintaining the internal covariant shift necessary for stable training.
ANN (Continuous): 0.72 --> 0.51 --> 0.39 --> 0.22 (Signal preserved)
SNN (Discrete): 1011 --> 0100 --> 0000 --> 0000 (Spikes vanish)
This compounding degradation manifests across several domains:
- Architectural Sparsity: When sparse binary spikes interact with sparse architectural components (such as attention heads or sparse convolutions), the signal can completely die out, causing neurons in deeper layers to stop firing entirely (Man Yao et al. in Spike2Former: Efficient Spiking Transformer for High-performance Image Segmentation; Xuerui Qiu et al. in Efficient 3D Recognition with Event-driven Spike Sparse Convolution).
- Temporal Constraints: In ultra-low latency settings, the processing window is restricted to very few timesteps. Neurons simply do not have enough time to accumulate sufficient membrane potential to reach their firing thresholds, preventing information from propagating through the entire architecture (Sayeed Shafayet Chowdhury et al. in Towards Ultra Low Latency Spiking Neural Networks for Vision and Sequential Tasks Using Temporal Pruning).
- Representation Loss: Compared to continuous-valued networks, SNNs exhibit a progressive loss of representational quality in deeper layers, meaning that the information bottleneck compounds layer by layer (Yuhan Zhang 0006 et al. in Enhancing Representation of Spiking Neural Networks via Similarity-Sensitive Contrastive Learning; Hongwei Ren et al. in SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition).
The Backward Pass: The Dormant Unit Dilemma
The vanishing spike problem in the forward pass directly cripples the network's ability to learn. Because the mathematical step function that generates a spike is non-differentiable (its derivative is zero everywhere except at the threshold, where it is infinite), SNNs rely on smooth surrogate gradients during backpropagation.
However, surrogate gradients are only active within a narrow window centered around the firing threshold. If a neuron's internal voltage falls outside this narrow range, its surrogate gradient becomes zero.
Surrogate Gradient Curve:
_.._ <-- Active Region (Gradients flow)
.' '.
______.' '______ <-- Inactive Region (Gradients = 0)
^
Firing Threshold
When forward spikes vanish, the network encounters the dormant unit problem: internal neuron states drift into saturated, inactive regions where surrogate gradients are zero, halting weight updates entirely (Lang Feng et al. in Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks; Yufei Guo et al. in RecDis-SNN: Rectifying Membrane Potential Distribution for Directly Training Spiking Neural Networks).
This gradient degradation is compounded by several factors:
- Sparsity Regularization: Aggressive pruning or regularization designed to lower energy consumption can trap the model in theoretical saddle points where neurons stop communicating and gradients vanish completely (Tong Bu et al. in Activity Pruning for Efficient Spiking Neural Networks).
- Gradient Bounding: Because surrogate functions are inherently bounded approximations, they lose critical gradient information as signals propagate backward through deep architectures (Yufei Guo et al. in Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks; Jiangrong Shen et al. in Learnable Surrogate Gradient for Direct Training Spiking Neural Networks).
- Gradient Leakage: Standard operations like average pooling distribute gradient signals across all units in a window—including inactive ones—diluting the learning signal (Wei Fang et al. in Training Spiking Neural Networks with Event-driven Backpropagation).
- Stochasticity Limits: Even when noise is introduced to smooth out the landscape, units that remain below the threshold fail to provide any gradient signal to optimize parameters (Christian Holberg et al. in Exact Gradients for Stochastic Spiking Neural Networks Driven by Rough Signals).
Solutions: Keeping the Signal Alive
To successfully train deep SNNs, researchers have developed mechanisms to preserve information flow in both the forward and backward passes. These solutions fall into three main categories: architectural shortcuts, state normalization, and threshold engineering.
Architectural Shortcuts and Residual Tracking
In a standard spiking neuron, the membrane potential is reset to a baseline value immediately after firing, discarding any excess voltage. To prevent this information loss, Bing Han et al. introduced the Residual Membrane Potential (RMP) neuron in Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network. By retaining the residual voltage after a firing event, the neuron preserves sub-threshold information across timesteps, mitigating signal attenuation in deep networks.
Standard Reset: Potential = 1.2 --> Fire Spike --> Potential = 0.0 (0.2 lost)
Residual Reset: Potential = 1.2 --> Fire Spike --> Potential = 0.2 (Preserved)
On a macro level, adapting residual connections (ResNets) to SNNs requires careful handling to avoid disrupting identity mappings. Wei Fang et al. designed the Spike-Element-Wise (SEW) ResNet in Deep Residual Learning in Spiking Neural Networks, which restructures the residual block to allow spikes and gradients to flow seamlessly across networks up to 152 layers deep without degradation.
State Normalization and Proper Initialization
Another way to prevent spikes from vanishing is to explicitly control the distribution of internal neuron states so they remain within the active zone of surrogate gradients.
The authors of Solving the spike feature information vanishing problem in spiking deep Q network with potential based normalization introduced Potential-based Layer Normalization (pbLN), which normalizes membrane potentials directly across layers to stabilize the internal covariant shift.
To achieve stability without the computational overhead of normalization layers during inference, Xinyu Shi et al. proposed SpikeInit in Training Deep Spiking Neural Networks without Normalization. This method mathematically derives optimal initial weight distributions to ensure that firing rates neither explode nor vanish, enabling the training of ultra-deep SNNs up to 1,000 layers. Similarly, Aurora Micheli et al. demonstrated in Deep activity propagation via weight initialization in spiking neural networks that accounting for the discrete nature of spike quantization during initialization prevents neural activity from dissipating as it propagates.
Threshold and Firing Engineering
Rather than forcing neurons to conform to a rigid binary threshold, some approaches modify the firing mechanism itself to widen the active gradient window.
In Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks, the authors introduced Multi-Level Firing (MLF). By allowing neurons to fire multiple spikes at graded threshold levels, they broaden the active range of the surrogate gradient, ensuring that deep layers continue to receive robust weight updates and eliminating the dormant unit problem entirely.
Key Foundational Work
Minibatch Processing for Speed-up and Scalability of Spiking Neural Network Simulation (2020) — contributed 2 concept elements to this direction.
RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network (2020) — contributed 1 concept elements to this direction.
Unsupervised Visual Feature Learning with Spike-timing-dependent Plasticity: How Far are we from Traditional Feature Learning Approaches? (2019) — contributed 1 concept elements to this direction.