Memory-Efficient Reversible Spiking Neural Networks

Hong Zhang, Yu Zhang

Introduction

Training deep Spiking Neural Networks (SNNs) is notoriously memory-intensive. Unlike standard Artificial Neural Networks (ANNs), SNNs must store states across both network depth and simulation time steps. Standard training using Backpropagation Through Time (BPTT) requires the network to be "unfolded" over $T$ simulation time steps, leading to a memory cost that scales linearly with both depth ($D$) and time ($T$). This bottleneck often restricts researchers to shallow networks or small batch sizes, hindering SNNs from achieving the scale of modern ANNs.

This paper introduces Reversible Spiking Neural Networks (RevSNNs), which allow intermediate activations to be recomputed during the backward pass. This approach slashes GPU memory usage by up to 8x, enabling the training of much deeper and more powerful SNNs without sacrificing accuracy.

The Reversible Spiking Block

The core innovation is a Reversible Spiking Block that adapts the concept of reversible architectures for the spiking domain.

Spatial Reversibility

The input $X$ is split into two halves, $X_1$ and $X_2$. The forward pass is defined as:

  • $Y_1 = X_1 + \mathcal{F}(X_2)$
  • $Y_2 = X_2 + \mathcal{G}(Y_1)$

Because of this structure, the inputs $X_1$ and $X_2$ can be mathematically recovered from the outputs $Y_1$ and $Y_2$ during the backward pass:

  • $X_2 = Y_2 - \mathcal{G}(Y_1)$
  • $X_1 = Y_1 - \mathcal{F}(X_2)$

This means intermediate states do not need to be cached in memory.

Forward and reverse passes of the Reversible Spiking Block across time steps.

Temporal Consistency and the Reset Requirement

While the block is reversible across layers (space), it must remain consistent across time steps. Spiking neurons possess internal "memory" in the form of membrane potentials. The authors demonstrate that as long as the spiking neurons are reset to their initial states before the reverse process, the recomputed variables are identical to those in the forward pass. This "reset process" is the critical step that enables reversibility in SNNs.

Key Insights

  • Decoupling Depth from Memory: In a standard SNN, adding layers increases memory linearly. In a reversible sequence, only the output of the last block needs to be stored, making memory cost largely independent of depth.
  • Inherent Regularization: Reversible models often achieve slightly higher accuracy than non-reversible counterparts, likely due to the stronger inherent regularization provided by the architecture.
  • Larger Batch Sizes: By saving memory, the authors increased the maximum batch size by over 2.6x, stabilizing training and improving final performance.

Experimental Results

The authors applied this logic to two state-of-the-art architectures: RevSResNet (based on Spiking ResNet) and RevSFormer (based on the Spiking Vision Transformer).

GPU memory consumption per image vs. model complexity (GFLOPs). Reversible models maintain nearly constant memory usage as depth increases.

The models were evaluated on static (CIFAR-10/100) and neuromorphic (CIFAR10-DVS, DVS128 Gesture) datasets. Key findings include:

  • Massive Memory Savings: RevSResNet37 consumed 3.79× less GPU memory per image than its non-reversible counterpart. For deeper models, savings reached up to 8.1× for ResNet-based architectures and 9.1× for Transformer-based ones.
  • Competitive Accuracy: On CIFAR-10, RevSResNet37 achieved 94.77% accuracy, and RevSFormer-4-384 reached 95.34%, matching or slightly exceeding non-reversible state-of-the-art models.

Limitations and Future Work

While highly effective, reversibility involves a trade-off. Training takes roughly 33% longer due to the additional recomputation step during the backward pass. Furthermore, not every component is reversible; downsampling layers and initial tokenizers still require standard memory caching. Future research may explore making the temporal dimension itself reversible to further reduce the $O(T)$ memory dependency.

Summary updated

Abstract

Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) due to their high energy-efficiency on neuromorphic hardware. However, SNNs are unfolded over simulation time steps during the training process. Thus, SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. In this paper, we propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training. Firstly, we extend the reversible architecture along temporal dimension and propose the reversible spiking block, which can reconstruct the computational graph and recompute all intermediate variables in forward pass with a reverse process. On this basis, we adopt the state-of-the-art SNN models to the reversible variants, namely reversible spiking ResNet (RevSResNet) and reversible spiking transformer (RevSFormer). Through experiments on static and neuromorphic datasets, we demonstrate that the memory cost per image of our reversible SNNs does not increase with the network depth. On CIFAR10 and CIFAR100 datasets, our RevSResNet37 and RevSFormer-4-384 achieve comparable accuracies and consume 3.79x and 3.00x lower GPU memory per image than their counterparts with roughly identical model complexity and parameters. We believe that this work can unleash the memory constraints in SNN training and pave the way for training extremely large and deep SNNs. The code is available at https://github.com/mi804/RevSNN.git.
BibTeX
@inproceedings{zhang2024memory,
  author = {Hong Zhang and Yu Zhang},
  title = {{Memory-Efficient Reversible Spiking Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i15.29616},
}
Figure