RTFormer: Re-parameter TSBN Spiking Transformer

Hongzhi Wang, Xiubo Liang, Mengjian Li, Tao Zhang

Abstract

Spiking Neural Networks (SNNs) offer a bio-inspired, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) by utilizing discrete binary spikes. However, balancing high-tier computational performance with the inherent efficiency of SNNs remains a significant challenge. This paper introduces RTFormer, a Spiking Transformer architecture designed to bridge this gap. By integrating a Spatial-Temporal Core (ST-Core) featuring structural re-parameterization and a novel Temporal Sliding Batch Normalization (TSBN), RTFormer achieves state-of-the-art accuracy on ImageNet while reducing energy consumption during inference.

Introduction

The quest for energy-efficient AI has led researchers toward Spiking Neural Networks (SNNs), which mimic the sparse, event-driven communication of the human brain. While SNNs are theoretically ideal for neuromorphic hardware, they often lag behind traditional Transformers in complex vision tasks. The primary hurdles include the difficulty of capturing complex spatial features without excessive computational overhead and the challenge of effectively normalizing time-dependent spiking data.

RTFormer addresses these issues through a dual-pronged approach: optimizing spatial feature extraction via structural re-parameterization and refining temporal dynamics with a sliding-window normalization technique. This results in a model that is powerful during training but lean and efficient during deployment.

Methodology: The Spatial-Temporal Core (ST-Core)

The innovation of RTFormer centers on the Spatial-Temporal Core (ST-Core), which optimizes the network across two critical dimensions: spatial representation and temporal dynamics.

1. The Spatial Core: Structural Re-parameterization

To capture rich spatial information, the model employs structural re-parameterization. During the training phase, the architecture utilizes a multi-branch convolutional structure, including 1x1 and 3x3 depthwise convolutions. This complexity allows the model to learn diverse feature representations.

Post-training, these multiple branches are mathematically collapsed into a single, streamlined 3x3 convolution. This "RepVGG-style" transformation ensures that the inference model retains the representational power of a large network while benefiting from the speed and low memory footprint of a simplified structure.

2. The Temporal Core: Temporal Sliding Batch Normalization (TSBN)

Standard Batch Normalization (BN) often fails to account for the temporal nuances of spiking data, either by averaging across all time steps or ignoring the temporal context entirely. RTFormer introduces Temporal Sliding Batch Normalization (TSBN), which utilizes a sliding window to capture local temporal dynamics. This approach allows the network to adapt to shifts in data over time more effectively than static methods.

3. Threshold Folding for Zero-Overhead Inference

A key efficiency gain in RTFormer is Threshold Folding. The parameters learned by the TSBN are integrated directly into the spiking neuron's firing threshold ($V_{th}$). By "folding" the normalization into the neuron's natural activation logic, the model eliminates the need for an explicit normalization step during inference. This hardware-aligned strategy ensures that the benefits of TSBN are realized without increasing the energy cost per spike.

Experimental Results

The researchers evaluated RTFormer against standard static image benchmarks and native neuromorphic datasets.

Performance on ImageNet and CIFAR

RTFormer demonstrated superior performance on the ImageNet-1K dataset, achieving a top-1 accuracy of 80.54%. This significantly outperforms previous SNN leaders like Spikformer (74.81%) and Spike-Driven Transformer (77.07%). On smaller datasets, RTFormer reached 96.27% on CIFAR-10 and 81.37% on CIFAR-100, establishing a new standard for Spiking Transformers.

Energy Efficiency

Energy consumption is a critical metric for SNNs. On ImageNet, RTFormer consumed only 5.59 mJ per inference. Notably, this is lower than the 6.09 mJ required by the Spike-Driven Transformer, despite RTFormer's higher accuracy. This result highlights the effectiveness of the ST-Core and threshold folding in minimizing computational waste.

Neuromorphic Data Versatility

To test its efficacy on event-based data, the model was evaluated on the DVS128 Gesture dataset. RTFormer achieved 98.61% accuracy, proving its ability to process native "spike trains" from dynamic vision sensors—a crucial requirement for real-world neuromorphic applications.

Conclusion and Future Work

RTFormer represents a significant step forward in making Spiking Transformers both smarter and leaner. By leveraging structural re-parameterization and temporal sliding normalization, the architecture achieves high-performance vision processing with industry-leading energy efficiency.

While the current results are impressive, future research could explore scaling RTFormer for more complex temporal tasks such as object detection or real-time video segmentation. Furthermore, verifying the architecture's performance across a wider variety of neuromorphic hardware platforms will be essential for widespread adoption in edge computing and robotics.

Summary updated

Abstract

The Spiking Neural Networks (SNNs), renowned for their bio-inspired operational mechanism and energy efficiency, mirror the human brain's neural activity. Yet, SNNs face challenges in balancing energy efficiency with the computational demands of advanced tasks. Our research introduces the RTFormer, a novel architecture that embeds Re-parameterized Temporal Sliding Batch Normalization (TSBN) within the Spiking Transformer framework. This innovation optimizes energy usage during inference while ensuring robust computational performance. The crux of RTFormer lies in its integration of reparameterized convolutions and TSBN, achieving an equilibrium between computational prowess and energy conservation.
BibTeX
@inproceedings{wang2024rtformer,
  author = {Hongzhi Wang and Xiubo Liang and Mengjian Li and Tao Zhang},
  title = {{RTFormer: Re-parameter TSBN Spiking Transformer}},
  booktitle = {IJCNN},
  year = {2024},
  url = {https://openreview.net/forum?id=eFRfnQH22y},
}
Figure