Introduction
Training deep Spiking Neural Networks (SNNs) is notoriously memory-intensive. Unlike standard Artificial Neural Networks (ANNs), SNNs must store states across both network depth and simulation time steps. Standard training using Backpropagation Through Time (BPTT) requires the network to be "unfolded" over $T$ simulation time steps, leading to a memory cost that scales linearly with both depth ($D$) and time ($T$). This bottleneck often restricts researchers to shallow networks or small batch sizes, hindering SNNs from achieving the scale of modern ANNs.
This paper introduces Reversible Spiking Neural Networks (RevSNNs), which allow intermediate activations to be recomputed during the backward pass. This approach slashes GPU memory usage by up to 8x, enabling the training of much deeper and more powerful SNNs without sacrificing accuracy.
The Reversible Spiking Block
The core innovation is a Reversible Spiking Block that adapts the concept of reversible architectures for the spiking domain.
Spatial Reversibility
The input $X$ is split into two halves, $X_1$ and $X_2$. The forward pass is defined as:
- $Y_1 = X_1 + \mathcal{F}(X_2)$
- $Y_2 = X_2 + \mathcal{G}(Y_1)$
Because of this structure, the inputs $X_1$ and $X_2$ can be mathematically recovered from the outputs $Y_1$ and $Y_2$ during the backward pass:
- $X_2 = Y_2 - \mathcal{G}(Y_1)$
- $X_1 = Y_1 - \mathcal{F}(X_2)$
This means intermediate states do not need to be cached in memory.
Temporal Consistency and the Reset Requirement
While the block is reversible across layers (space), it must remain consistent across time steps. Spiking neurons possess internal "memory" in the form of membrane potentials. The authors demonstrate that as long as the spiking neurons are reset to their initial states before the reverse process, the recomputed variables are identical to those in the forward pass. This "reset process" is the critical step that enables reversibility in SNNs.
Key Insights
- Decoupling Depth from Memory: In a standard SNN, adding layers increases memory linearly. In a reversible sequence, only the output of the last block needs to be stored, making memory cost largely independent of depth.
- Inherent Regularization: Reversible models often achieve slightly higher accuracy than non-reversible counterparts, likely due to the stronger inherent regularization provided by the architecture.
- Larger Batch Sizes: By saving memory, the authors increased the maximum batch size by over 2.6x, stabilizing training and improving final performance.
Experimental Results
The authors applied this logic to two state-of-the-art architectures: RevSResNet (based on Spiking ResNet) and RevSFormer (based on the Spiking Vision Transformer).
The models were evaluated on static (CIFAR-10/100) and neuromorphic (CIFAR10-DVS, DVS128 Gesture) datasets. Key findings include:
- Massive Memory Savings: RevSResNet37 consumed 3.79× less GPU memory per image than its non-reversible counterpart. For deeper models, savings reached up to 8.1× for ResNet-based architectures and 9.1× for Transformer-based ones.
- Competitive Accuracy: On CIFAR-10, RevSResNet37 achieved 94.77% accuracy, and RevSFormer-4-384 reached 95.34%, matching or slightly exceeding non-reversible state-of-the-art models.
Limitations and Future Work
While highly effective, reversibility involves a trade-off. Training takes roughly 33% longer due to the additional recomputation step during the backward pass. Furthermore, not every component is reversible; downsampling layers and initial tokenizers still require standard memory caching. Future research may explore making the temporal dimension itself reversible to further reduce the $O(T)$ memory dependency.
Summary updated