Abstract: Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) due to their high energy-efficiency on neuromorphic hardware. However, SNNs are unfolded over simulation time steps during the training process. Thus, SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. In this paper, we propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training. Firstly, we extend the reversible architecture along temporal dimension and propose the reversible spiking block, which can reconstruct the computational graph and recompute all intermediate variables in forward pass with a reverse process. On this basis, we adopt the state-of-the-art SNN models to the reversible variants, namely reversible spiking ResNet (RevSResNet) and reversible spiking transformer (RevSFormer). Through experiments on static and neuromorphic datasets, we demonstrate that the memory cost per image of our reversible SNNs does not increase with the network depth. On CIFAR10 and CIFAR100 datasets, our RevSResNet37 and RevSFormer-4-384 achieve comparable accuracies and consume 3.79x and 3.00x lower GPU memory per image than their counterparts with roughly identical model complexity and parameters. We believe that this work can unleash the memory constraints in SNN training and pave the way for training extremely large and deep SNNs.
The Memory Wall in Spiking Neural Networks
Spiking Neural Networks (SNNs) are often hailed as the "third generation" of neural networks, prized for their biological plausibility and energy efficiency on neuromorphic hardware. However, training them on standard GPUs presents a significant challenge. Unlike standard Artificial Neural Networks (ANNs), SNNs process information over several simulation time steps ($T$). When training with Backpropagation Through Time (BPTT), the GPU must store intermediate activations and membrane potentials for every layer at every time step.
This requirement causes memory usage to scale linearly with both network depth ($D$) and time ($T$). Consequently, researchers often hit a "memory wall," forced to use small batch sizes or shallow networks. This limitation has historically hampered the final accuracy and scalability of SNNs compared to their ANN counterparts.
Methods: The Spiking Reversible Block
To break this bottleneck, this research introduces the Spiking Reversible Block. The core concept is inspired by reversible architectures (RevNets): if the input of a layer can be mathematically reconstructed from its output, there is no need to store intermediate states during the forward pass.
Key Innovations
- Two-Stream Architecture: The input is split into two halves, $X_1$ and $X_2$. These are updated in a staggered fashion using two residual functions ($F$ and $G$). This structure allows the forward operations to be inverted during the backward pass.
- Temporal Consistency: SNNs have internal states (membrane potentials) that evolve over time. The proposed block is designed to be reversible in the spatial dimension (layer to layer) while maintaining consistency across the temporal dimension.
- The Reset-Reverse Process: To ensure exact reconstruction, the authors introduce a "reset" step. Before the backward pass, neurons are reset to their initial states. The network then recomputes the forward pass "on the fly" during backpropagation to provide the necessary gradients.
Experimental Results
The researchers evaluated their models, RevSResNet and RevSFormer, on both static (CIFAR-10/100) and neuromorphic (CIFAR10-DVS, DVS128 Gesture) datasets.
Decoupling Depth from Memory
The most significant finding is that for a reversible sequence, the memory cost per image becomes independent of the number of blocks. As shown in the experimental data, while standard SNN memory usage spikes as layers are added, the RevSNN memory cost remains remarkably flat.
Performance Highlights
- Memory Efficiency: On CIFAR-100, RevSResNet37 consumed 3.79x less GPU memory per image than the standard MS ResNet34. Similarly, RevSFormer achieved a 3.00x reduction in memory.
- Scaling Up: The memory savings allowed for significantly larger batch sizes. On a 24GB RTX 3090, RevSResNet37 could handle a batch size 2.69x larger than the non-reversible version.
- Accuracy: Accuracy remained competitive and, in some cases, slightly improved. For example, RevSResNet37 reached 94.77% on CIFAR-10, slightly outperforming the non-reversible baseline of 94.69%. This suggests that reversible architectures may provide a form of inherent regularization.
Limitations and Future Outlook
While RevSNN successfully decouples memory from network depth, some challenges remain:
- Temporal Scaling: Memory cost still increases with the number of simulation time steps ($T$) because the output across all time steps must be cached for the final classification head.
- Computational Overhead: Reversibility increases training time by approximately 33% due to the extra computations required during the backward pass.
- Downsampling: Currently, downsampling layers (which change feature map size) are not reversible and still require standard memory caching.
Despite these trade-offs, RevSNN represents a major step forward in SNN research, providing the tools necessary to train extremely deep and large-scale bio-inspired models.
Summary updated