Spiking Neural Networks (SNNs) are highly praised for their biological plausibility and energy efficiency. By processing information using sparse, binary "spikes" over time, they can theoretically run on neuromorphic hardware at a fraction of the power required by traditional Artificial Neural Networks (ANNs).
However, training SNNs on modern accelerators like GPUs is notoriously slow. The root cause is time itself. Traditional SNN simulations treat time as a sequential loop ($O(T)$ complexity). To compute a neuron's state at time step $t$, you must first compute its state at $t-1$ because of how membrane potentials accumulate and reset. This serial dependency creates a massive bottleneck for Backpropagation Through Time (BPTT), leaving powerful, highly parallel GPUs idling while waiting for the next time step.
To break this bottleneck, a rapidly growing body of research focuses on parallel temporal processing. Instead of stepping through time sequentially, these approaches reformulate spiking dynamics so that an entire sequence of time steps can be processed simultaneously. This shifts the computational complexity from linear ($O(T)$) to constant ($O(1)$ or $O(K)$) relative to sequence length, unlocking 10x to 100x training speedups.
Reshaping Tensors and Merging Dimensions
The most direct way to introduce parallelism is to restructure how data flows through the network's linear layers (such as convolutions or fully connected layers). In a standard SNN, a layer processes a batch of inputs at a single time step, updates its internal states, and repeats for the next time step.
To bypass this, frameworks like SpykeTorch (Milad Mozafari et al.) and SpikingJelly (Wei Fang et al.) represent spikes as high-dimensional tensors that explicitly include the time dimension (e.g., $\text{Time} \times \text{Batch} \times \text{Channel} \times \text{Height} \times \text{Width}$). By merging or swapping the time and batch dimensions, standard, highly optimized GPU operations can process all time steps simultaneously.
As Zecheng Hao et al. demonstrate, treating the temporal dimension of a neuron as a vector rather than a sequence eliminates the requirement to wait for previous time steps during linear transformations, dramatically reducing inference latency. Similarly, Jindong Li et al. design hardware accelerators around a four-dimensional parallel dataflow that integrates space, channels, and time steps to generate spikes on the fly, minimizing the need to store massive volumes of membrane potential data on-chip.
Linearizing Dynamics via State-Space Models
While merging dimensions parallelizes the synaptic inputs (the linear part of the network), the internal state update of a spiking neuron (the leaky integration) remains inherently sequential. For a classic Leaky Integrate-and-Fire (LIF) neuron, the membrane potential $V$ decays and accumulates over time:
$$V_t = \alpha V_{t-1} + X_t - V_{\text{th}} S_{t-1}$$
The presence of the hard reset—subtracting the threshold voltage $V_{\text{th}}$ when a spike $S_{t-1}=1$ occurs—makes this recurrence highly non-linear and strictly sequential.
However, if the reset mechanism is modified or omitted, the neuron's dynamics simplify into a Linear Time-Invariant (LTI) system or a State Space Model (SSM). As Malyaban Bal et al. and Kexin Wang et al. show, an LTI recurrence can be mathematically unrolled into a single convolutional operation across the entire time series. This allows the entire sequence of hidden states to be computed simultaneously using Fast Fourier Transforms (FFTs).
By removing the traditional reset or refractory periods, architectures like the Parallel Spiking Neuron (PSN) proposed by Wei Fang et al. reformulate temporal integration as a massive matrix-matrix multiplication. Similarly, Thomas E. Huber et al. scale Resonate-and-Fire neurons by mapping them to structured state-space models, enabling parallel scan operations. This mechanism decouples the temporal sequence during training to achieve constant-time complexity relative to the input length, as noted by Chenxiang Ma et al..
Conquering the Non-Linear Reset Bottleneck
Completely removing the reset mechanism sacrifices some of the biological realism and sparse, event-driven nature of SNNs. To maintain a strict, non-linear threshold reset while still processing time in parallel, researchers have introduced three clever mathematical formulations:
- Fixed-Point Iteration: Wanjin Feng et al. reformulate the entire temporal sequence of a neuron's states as a fixed-point problem. Instead of solving the equations step-by-step, the GPU guesses the states across all time steps simultaneously and iteratively refines them. Because this process converges in a small, constant number of iterations ($K \approx 3$), it reduces training complexity from $O(T)$ to $O(K)$, yielding up to a 100x speedup on long-duration tasks.
- Boundary Compression: Yan Zhong et al. treat spike determination as a boundary problem. They use FFTs to compute the cumulative inputs across a sequence first, and then iteratively compress the upper and lower bounds of the membrane potential to resolve all firing events in parallel across the entire sequence at once.
- Speculative Execution: Todd Morrill et al. introduce a chunk-based approach. The system speculatively calculates future states in parallel assuming no spikes occur. If a spike is detected within a chunk, the invalid downstream states are discarded, and the system restarts from the reset point using parallel associative scans and differentiable root solvers.
Decoupling Gradients in the Backward Pass
Even if the forward pass is parallelized, backpropagation through time (BPTT) traditionally requires tracking gradients sequentially backward through every discrete step, which is highly memory-intensive.
To solve this, Jingtao Wang et al. introduce Stabilized Spiking Flow (SSF). Instead of tracking discrete spike sequences during the backward pass, they approximate spike trains as steady-state continuous flows of average input and output activations. This removes the serial dependency of membrane potential accumulation during gradient updates, allowing the backward pass to be completely decoupled and parallelized across hardware.
Sequential BPTT (Traditional):
Step T --------> Step T-1 --------> Step T-2 --------> Step 1 (Strictly Serial)
Stabilized Spiking Flow (Parallel):
[Step T] [Step T-1] [Step T-2] [Step 1]
| | | |
v v v v (Simultaneous Updates)
------------------- Parallelized Gradient Flow -------------------
Furthermore, this parallel-to-serial decoupling can be asymmetric. As Dehao Zhang et al. and Zecheng Hao et al. demonstrate, networks can be trained using highly parallelized temporal structures on GPUs, and then mathematically converted back into serial, event-driven processes for deployment on edge neuromorphic hardware. This bridges the gap between the parallel processing strengths of modern training hardware and the sequential, energy-efficient nature of biological modeling.
Key Foundational Work
SpykeTorch: Efficient Simulation of Convolutional Spiking Neural Networks With at Most One Spike per Neuron (2019) — contributed 3 concept elements to this direction.
Implementation of Bayesian Inference In Distributed Neural Networks (2018) — contributed 1 concept elements to this direction.