Identity Mapping for Deep Spiking Neural Networks

63 concepts37 papers1 proposal

Spiking Neural Networks (SNNs) are highly attractive for energy-efficient computing because they communicate via discrete, binary impulses (spikes) rather than continuous values. On neuromorphic hardware, this sparse, event-driven signaling replaces power-hungry matrix multiplications with simple additions.

However, training deep SNNs directly via backpropagation has historically been a major challenge. Because a spike is generated by a discrete threshold function, its derivative is zero everywhere and undefined at the threshold. While researchers bypass this using continuous surrogate gradients during the backward pass, scaling SNNs to deeper architectures introduces a severe degradation problem: gradients either vanish or explode exponentially as they propagate through chains of non-differentiable layers.

In standard Artificial Neural Networks (ANNs), this degradation is solved by residual connections (ResNets) that create an identity mapping:

$$y = x + F(x)$$

This formulation ensures that the gradient with respect to the input contains a clean term of unity ($\frac{\partial y}{\partial x} = 1 + \frac{\partial F}{\partial x}$), allowing error signals to flow backward unimpeded even if the residual branch $F(x)$ vanishes. In SNNs, however, simply adding spikes or passing them through standard blocks breaks this identity property. A dedicated line of research has emerged to redesign the mathematical and structural flow of spiking residual blocks to achieve true identity mapping.


Shifting the Activation: Spike-Element-Wise ResNets

In a conventional residual block, the shortcut connection and the residual branch are summed before the activation function: $\text{ReLU}(x + F(x))$. If we directly translate this topology to an SNN, the summation of the inputs passes through a hard spiking threshold function:

$$y = \text{Spike}(x + F(x))$$

When taking the derivative, the gradient becomes proportional to the derivative of the spiking function: $\text{Spike}'(x + F(x)) \cdot (1 + \frac{\partial F}{\partial x})$. Because the surrogate gradient $\text{Spike}'$ is a narrow approximation that quickly drops to zero outside a small window, the crucial "$1$" that preserves the identity mapping is multiplied by zero or a fraction. The gradient highway is effectively blocked.

To solve this, Fang et al. introduced the Spike-Element-Wise (SEW) ResNet. The core intuition is to shift the element-wise addition to a point after the spiking activation functions:

$$y = \text{Spike}(x) + \text{Spike}(F(x))$$

By performing element-wise operations between the discrete outputs of the spiking neurons, the gradient during identity mapping remains exactly one when the residual branch is inactive. This architectural modification prevents the error signal from being attenuated by the surrogate gradients of the spiking neurons, allowing SNNs to scale to over 150 layers while consistently reducing training loss.

A similar structural principle is leveraged in the Spiking DS-ResNet architecture discussed in Multi-Level Firing with Spiking DS-ResNet: Enabling Better and Deeper Directly-Trained Spiking Neural Networks. By placing the spiking activation function before the shortcut addition, the network suppresses the formation of "dormant units" (neurons that stop firing and freeze learning) and preserves the mathematical identity mapping across deep layers.


Bypassing Spikes: Membrane Potential Residuals

An alternative approach to maintaining gradient stability is to avoid routing the shortcut connection through the discrete spike generation mechanism altogether. Instead of summing binary spikes, these architectures establish residual connections across the continuous internal states of the neurons—their membrane potentials.

As explored by Qiu et al. and Fang et al., connecting the internal membrane potentials across different layers creates continuous voltage shortcuts. Because the membrane potential accumulates analog values over time before a spike is triggered, it remains a differentiable state.

By allowing the accumulated voltage to bypass standard activation functions, the network ensures that information and gradients persist through multiple layers without losing spike density or suffering from signal degradation. This design choice maintains high representational capacity even when the network scales to complex 3D vision or transformer topologies.


Gated and Normalized Identity Paths

Maintaining a perfect identity mapping can sometimes conflict with the goal of keeping spike distributions sparse and stable. If spikes accumulate uncontrollably along the shortcut paths, the network suffers from exploding spike activity.

To balance gradient flow with binary signal integrity, Zhang et al. proposed the Spiking Gate (SG) ResNet. This architecture introduces a learnable binary selection gate that acts as a switch between the identity mapping and the spiking neuron's output:

               +-----------------------+
               |                       |  (Identity Path)
               v                       v
Input x ---> [ Layer F(x) ] ---> [ Binary Gate ] ---> Output y

This mechanism allows the network to dynamically choose when to preserve the identity gradient and when to enforce a sparse spiking representation, enabling deep scaling without performance loss.

Complementary to gating, normalization layers can be adapted to stabilize these paths. Zheng et al. developed threshold-dependent batch normalization (tdBN), which integrates normalization steps directly into the spiking shortcut paths. This ensures that when the signals from the identity and residual branches are combined, their joint distribution remains bounded, preventing the vanishing or exploding spike activity common in direct translations of standard architectures (Direct learning-based deep spiking neural networks: a review).


Mathematical Alignment with Continuous Networks

Rather than altering the network topology, can we make the spiking neuron itself inherently support identity mapping during backpropagation?

Stanojević demonstrated that the gradient instability problem can be solved by mathematically aligning the training trajectories of SNNs with standard continuous networks. By forcing the gradient of a spiking neuron's membrane potential to remain constant at the firing threshold, the author aligned the Jacobian of the spiking network with that of a standard Rectified Linear Unit (ReLU) network:

$$\text{Jacobian}_{\text{SNN}} \equiv \text{Jacobian}_{\text{ReLU}}$$

This exact mathematical alignment ensures that the gradient descent trajectory of the spiking model behaves identically to its continuous counterpart, eliminating the compounding scaling issues that cause gradients to vanish or explode in deep architectures.


Temporal Identity Mappings in Recurrent SNNs

The challenge of gradient stability in SNNs is not only spatial (across layers) but also temporal (across simulation time steps). When training recurrent SNNs using Backpropagation Through Time (BPTT), sequential multiplications of the spiking threshold derivative cause gradients to vanish over long temporal horizons.

To establish temporal identity mappings, researchers have turned to multi-compartment neuron designs and local memory loops. Zhang et al. and Zhang et al. introduced a two-compartment spiking neuron model (TC-LIF) that mimics the interaction between a biological neuron's dendrites and its soma. By carefully initializing the coupling coefficients between these compartments, the authors proved that the network can maintain a temporal gradient norm above unity:

$$\| \nabla_{\text{time}} \| \ge 1$$

This mathematical property prevents the exponential decay of learning signals over hundreds of time steps.

Similarly, Liu et al. introduced the LIF Recurrent Memory Module (LRMM), which integrates local recurrent loops within the spiking micro-circuit. These loops act as a dedicated hardware-friendly memory highway, providing a stable path for gradients during BPTT and allowing the network to learn long-term temporal dependencies that are otherwise unreachable for standard leaky integrate-and-fire models.

Key Foundational Work

Deep Residual Learning in Spiking Neural Networks (2021) — contributed 5 concept elements to this direction.
Deep Residual Learning in Spiking Neural Networks (2021) — contributed 1 concept elements to this direction.
Going Deeper With Directly-Trained Larger Spiking Neural Networks (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_stability_in_spiking_networks_5a0d6224,
  title = {gradient stability in spiking networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-stability-in-spiking-networks/rdr_5a0d6224235f42249da5787c07073c5a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_stability_in_spiking_networks_5a0d6224_v1,
  title = {gradient stability in spiking networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-stability-in-spiking-networks/rdr_5a0d6224235f42249da5787c07073c5a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake