RTFormer: Re-parameter TSBN Spiking Transformer

Hongzhi Wang, Xiubo Liang, Mengjian Li, Tao Zhang

TL;DR

RTFormer is a novel Spiking Transformer architecture that achieves state-of-the-art accuracy (80.54% on ImageNet) while significantly slashing energy consumption. It achieves this by combining structural re-parameterization for efficient spatial processing with a new Temporal Sliding Batch Normalization (TSBN) that captures local temporal dynamics without adding overhead during inference.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are the "brain-inspired" cousins of traditional Artificial Neural Networks (ANNs). Instead of continuous values, they communicate via discrete binary "spikes," making them incredibly energy-efficient on specialized neuromorphic hardware. However, SNNs face a "performance-efficiency" gap:

  1. Complexity vs. Efficiency: High-performing architectures like Transformers are computationally heavy.
  2. Temporal Dynamics: Standard normalization techniques (like Batch Norm) often struggle to handle the time-dependent nature of spiking data effectively.
  3. Inference Overhead: Many methods that improve training accuracy add complexity that drains battery life during actual use (inference).

The authors of RTFormer set out to design a structure that maintains the high accuracy of Transformers while ensuring the inference process remains as energy-frugal as possible.

Methods

The core of the paper is the Spatial-Temporal Core (ST-Core), which splits the optimization into two dimensions:

  • The Spatial Core (Structural Re-parameterization): During training, the model uses a complex, multi-branch convolutional structure (including 1x1 and 3x3 depthwise convolutions) to capture rich spatial features. Once training is complete, these branches are mathematically fused into a single, streamlined 3x3 convolution. This "RepVGG-style" approach gives the model the representational power of a large network during training but the speed of a small one during inference.
  • The Temporal Core (TSBN): The authors introduce Temporal Sliding Batch Normalization (TSBN). Unlike previous methods that either normalize across all time steps or treat each step in isolation, TSBN uses a sliding window. This allows the model to adapt to local temporal shifts in data.
  • Threshold Folding: Crucially, the parameters of the TSBN are "folded" directly into the spiking neuron's firing threshold ($V_{th}$). This means the normalization happens "for free" as part of the neuron's natural spiking logic, eliminating the need for an explicit normalization step during inference.

Datasets

The researchers evaluated RTFormer on two types of data:

  • Static Image Datasets: ImageNet (1.3M images), CIFAR-10, and CIFAR-100. These are standard benchmarks for computer vision.
  • Neuromorphic (Event-based) Datasets: CIFAR10-DVS and DVS128 Gesture. These datasets consist of "spike trains" captured by dynamic vision sensors, which are much closer to how biological eyes and neuromorphic hardware operate.

Key Insights and Ideas

  • Reparameterization is a "Free Lunch" for SNNs: By using structural re-parameterization, the authors prove you can have a complex training regime that results in a very simple, energy-efficient inference model.
  • Local Context Matters in Time: The success of the sliding window (TSBN) suggests that for SNNs, the immediate temporal neighborhood of a spike is more informative for normalization than the entire sequence or a single isolated moment.
  • Hardware-Algorithm Synergy: By folding normalization into the membrane potential threshold, the authors align the software architecture with the way neuromorphic chips (like Loihi or Tianjic) actually function, maximizing the benefits of the hardware.

Key Results

RTFormer sets new benchmarks across the board:

  • ImageNet: Achieved 80.54% top-1 accuracy, outperforming the previous state-of-the-art Spikformer (74.81%) and Spike-Driven Transformer (77.07%).
  • Energy Efficiency: On ImageNet, RTFormer consumed only 5.59 mJ per inference, which is lower than its less-accurate competitors (e.g., Spike-Driven Transformer at 6.09 mJ).
  • Neuromorphic Superiority: On the DVS128 Gesture dataset, it reached 98.61% accuracy, showing its versatility in handling native "spike" data from event-based cameras.
  • CIFAR Performance: Reached 96.27% on CIFAR-10 and 81.37% on CIFAR-100.

Limitations and Future Work

While the results are impressive, the paper focuses primarily on image classification. The authors do not discuss how this architecture would scale to more complex tasks like object detection or real-time video segmentation, where temporal windows might need to be significantly larger. Additionally, while the energy estimates are based on 45nm CMOS technology benchmarks, actual deployment on diverse neuromorphic hardware (which varies in how it handles depthwise convolutions) remains an area for practical verification.

Summary updated

Abstract

The Spiking Neural Networks (SNNs), renowned for their bio-inspired operational mechanism and energy efficiency, mirror the human brain’s neural activity. Yet, SNNs face challenges in balancing energy efficiency with the computational demands of advanced tasks. Our research introduces the RTFormer, a novel architecture that embeds Re-parameterized Temporal Sliding Batch Normalization (TSBN) within the Spiking Transformer framework. This innovation optimizes energy usage during inference while ensuring robust computational performance. The crux of RTFormer lies in its integration of reparameterized convolutions and TSBN, achieving an equilibrium between computational prowess and energy conservation. Our experimental results highlight its effectiveness, with RTFormer achieving notable accuracy on standard datasets like ImageNet (80.54%), CIFAR-10 (96.27%), and CIFAR-100 (81.37%), and excelling in neuromorphic datasets such as CIFAR10-DVS (83.6%) and DVS128 (98.61%). These achievements illustrate RTFormer’s versatility and establish its potential in the realm of energy-efficient neural computing.
BibTeX
@inproceedings{wang2024rtformer,
  author = {Hongzhi Wang and Xiubo Liang and Mengjian Li and Tao Zhang},
  title = {{RTFormer: Re-parameter TSBN Spiking Transformer}},
  booktitle = {IJCNN},
  year = {2024},
  url = {https://openreview.net/forum?id=eFRfnQH22y},
}

Loading related content...

Figure