Abstract
Spiking Neural Networks (SNNs) offer a bio-inspired, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) by utilizing discrete binary spikes. However, balancing high-tier computational performance with the inherent efficiency of SNNs remains a significant challenge. This paper introduces RTFormer, a Spiking Transformer architecture designed to bridge this gap. By integrating a Spatial-Temporal Core (ST-Core) featuring structural re-parameterization and a novel Temporal Sliding Batch Normalization (TSBN), RTFormer achieves state-of-the-art accuracy on ImageNet while reducing energy consumption during inference.
Introduction
The quest for energy-efficient AI has led researchers toward Spiking Neural Networks (SNNs), which mimic the sparse, event-driven communication of the human brain. While SNNs are theoretically ideal for neuromorphic hardware, they often lag behind traditional Transformers in complex vision tasks. The primary hurdles include the difficulty of capturing complex spatial features without excessive computational overhead and the challenge of effectively normalizing time-dependent spiking data.
RTFormer addresses these issues through a dual-pronged approach: optimizing spatial feature extraction via structural re-parameterization and refining temporal dynamics with a sliding-window normalization technique. This results in a model that is powerful during training but lean and efficient during deployment.
Methodology: The Spatial-Temporal Core (ST-Core)
The innovation of RTFormer centers on the Spatial-Temporal Core (ST-Core), which optimizes the network across two critical dimensions: spatial representation and temporal dynamics.
1. The Spatial Core: Structural Re-parameterization
To capture rich spatial information, the model employs structural re-parameterization. During the training phase, the architecture utilizes a multi-branch convolutional structure, including 1x1 and 3x3 depthwise convolutions. This complexity allows the model to learn diverse feature representations.
Post-training, these multiple branches are mathematically collapsed into a single, streamlined 3x3 convolution. This "RepVGG-style" transformation ensures that the inference model retains the representational power of a large network while benefiting from the speed and low memory footprint of a simplified structure.
2. The Temporal Core: Temporal Sliding Batch Normalization (TSBN)
Standard Batch Normalization (BN) often fails to account for the temporal nuances of spiking data, either by averaging across all time steps or ignoring the temporal context entirely. RTFormer introduces Temporal Sliding Batch Normalization (TSBN), which utilizes a sliding window to capture local temporal dynamics. This approach allows the network to adapt to shifts in data over time more effectively than static methods.
3. Threshold Folding for Zero-Overhead Inference
A key efficiency gain in RTFormer is Threshold Folding. The parameters learned by the TSBN are integrated directly into the spiking neuron's firing threshold ($V_{th}$). By "folding" the normalization into the neuron's natural activation logic, the model eliminates the need for an explicit normalization step during inference. This hardware-aligned strategy ensures that the benefits of TSBN are realized without increasing the energy cost per spike.
Experimental Results
The researchers evaluated RTFormer against standard static image benchmarks and native neuromorphic datasets.
Performance on ImageNet and CIFAR
RTFormer demonstrated superior performance on the ImageNet-1K dataset, achieving a top-1 accuracy of 80.54%. This significantly outperforms previous SNN leaders like Spikformer (74.81%) and Spike-Driven Transformer (77.07%). On smaller datasets, RTFormer reached 96.27% on CIFAR-10 and 81.37% on CIFAR-100, establishing a new standard for Spiking Transformers.
Energy Efficiency
Energy consumption is a critical metric for SNNs. On ImageNet, RTFormer consumed only 5.59 mJ per inference. Notably, this is lower than the 6.09 mJ required by the Spike-Driven Transformer, despite RTFormer's higher accuracy. This result highlights the effectiveness of the ST-Core and threshold folding in minimizing computational waste.
Neuromorphic Data Versatility
To test its efficacy on event-based data, the model was evaluated on the DVS128 Gesture dataset. RTFormer achieved 98.61% accuracy, proving its ability to process native "spike trains" from dynamic vision sensors—a crucial requirement for real-world neuromorphic applications.
Conclusion and Future Work
RTFormer represents a significant step forward in making Spiking Transformers both smarter and leaner. By leveraging structural re-parameterization and temporal sliding normalization, the architecture achieves high-performance vision processing with industry-leading energy efficiency.
While the current results are impressive, future research could explore scaling RTFormer for more complex temporal tasks such as object detection or real-time video segmentation. Furthermore, verifying the architecture's performance across a wider variety of neuromorphic hardware platforms will be essential for widespread adoption in edge computing and robotics.
Summary updated