Spiking Neural Networks (SNNs) are often celebrated for their energy efficiency and biological plausibility, yet they are notoriously difficult to train. Unlike traditional Artificial Neural Networks (ANNs), which process continuous values in a single forward pass, SNNs operate over discrete time steps, accumulating information in a hidden state known as the membrane potential. This temporal dimension introduces a unique challenge: temporal covariate shift.
As a network processes a sequence, the distribution of neural inputs and internal states shifts across time steps. This "moving target" makes the optimization landscape rugged and prevents deep architectures from converging. To solve this, researchers have developed specialized normalization techniques that align the network's internal dynamics with its discrete firing requirements.
The Problem of Temporal Covariate Shift
In standard deep learning, Batch Normalization (BN) stabilizes training by ensuring that the inputs to each layer have a consistent mean and variance. However, Duan et al. observed that in SNNs, the input distributions change not just across layers, but across time. If a model uses a single set of normalization statistics for all time steps, it fails to account for the evolving nature of the spike train.
This instability is compounded by the neuron's memory. As Guo et al. point out, even if the inputs to a neuron are normalized, the integration of residual membrane potential from previous time steps can disrupt the distribution. This creates an "internal covariate shift" within the neuron itself, causing the membrane potential to drift away from the firing threshold. When the potential drifts too far, gradients either vanish (if the neuron never fires) or explode (if it fires constantly), a phenomenon Liu et al. describe as temporal gradient misalignment.
Aligning Distributions with Firing Thresholds
The most effective solution to this instability is to normalize the membrane potential directly before the firing step. The intuition is simple: a spiking neuron only communicates when its potential crosses a specific threshold. Therefore, the goal of normalization should be to keep the distribution of the membrane potential in a "sweet spot" relative to that threshold.
Zheng et al. introduced threshold-dependent batch normalization (tdBN) to achieve this. By calculating statistics across both spatial and temporal dimensions, tdBN scales the pre-synaptic inputs so that the variance of the membrane potential is precisely tuned to the firing threshold. This ensures that neurons remain sensitive to their inputs without becoming over-saturated or permanently silent. Similarly, Zeng et al. proposed potential-based normalization to ensure the post-synaptic potential maintains a zero mean and unit variance, preventing the "spike vanishing" problem that typically plagues deep spiking architectures.
Synchronizing Normalization with Temporal Memory
Because spiking neurons are dynamical systems, their state at any given moment is a product of their history. Standard normalization methods that use global temporal statistics are "non-causal"—they use information from the future to normalize the present.
To address this, Jiang et al. developed Temporal Accumulated Batch Normalization (TAB). Instead of using a global average, TAB calculates statistics using an expanding window that accumulates data from the start of the sequence up to the current time step. This aligns the normalization process with the neuron's own temporal integration, effectively eliminating temporal covariate shift while allowing the network to function with extremely low latency.
For tasks requiring local sensitivity, Wang et al. proposed Temporal Sliding Batch Normalization (TSBN). By capturing dynamics within a moving window, the model can adapt to short-term shifts in spike patterns, which is particularly useful for transformers and other architectures that rely on capturing complex temporal dependencies.
Scaling to Deep Architectures
The shift toward spatio-temporal normalization has fundamentally changed the scalability of SNNs. Early spiking models were limited to shallow layers and toy datasets. By implementing these specialized regulation mechanisms:
- Wu et al. demonstrated that spatial activity normalization via auxiliary neurons can prevent activity explosion in deep networks.
- Zheng et al. successfully trained SNNs up to 50 layers deep, achieving competitive results on ImageNet.
- Xu et al. extended these concepts to graph-based data, showing that normalizing across feature and temporal dimensions allows for sparse, energy-efficient graph learning.
By treating time as a core component of the normalization process rather than an afterthought, these methods bridge the gap between the continuous world of gradient-based optimization and the discrete, temporal world of biological computation.
Key Foundational Work
Direct Training for Spiking Neural Networks: Faster, Larger, Better (2019) — contributed 3 concept elements to this direction.
Noisy Spiking Neurons with Temporal Coding have more Computational Power than Sigmoidal Neurons (1996) — contributed 1 concept elements to this direction.