Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification

Qingyu Wang, Duzhen Zhang, Tielin Zhang, Bo Xu

Abstract

Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer, whereby the Spiking Self-Attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this paper, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier Transform, Wavelet Transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies ($0.4\%$-$1.5\%$), higher running speed ($9\%$-$51\%$ for training and $19\%$-$70\%$ for inference), reduced theoretical energy consumption ($20\%$-$25\%$), and reduced GPU memory usage ($4\%$-$26\%$), compared to the standard spikformer.

Introduction

Spiking Neural Networks (SNNs) represent the "third generation" of neural networks, designed to mimic the brain's energy efficiency by communicating via discrete, binary "spikes." Recently, "Spikformers" have integrated the powerful Transformer architecture into SNNs using Spiking Self-Attention (SSA). However, SSA inherits a major flaw from standard Transformers: quadratic complexity ($O(N^2)$). As the input sequence length ($N$) grows, the computational cost explodes. Furthermore, the traditional "Query-Key-Value" similarity matching in attention may be redundant for sparse spike data, where correlations are often too weak to justify the heavy mathematical machinery of vanilla self-attention.

Methods: From Dynamic to Fixed Bases

The researchers propose a shift in perspective: self-attention is essentially a method of transforming information using a set of basis functions. In standard attention, these basis functions are "dynamic"—calculated on the fly from the input (Query $\times$ Key). The FWformer replaces these dynamic bases with fixed basis functions from classical signal processing:

  • Fourier Transform (FT): Uses fixed triangular (sine/cosine) bases to move data into the frequency domain.
  • Wavelet Transform (WT): Uses "wavelets" (like the Haar wavelet) to capture local features in the spatial domain.
The FWformer architecture replaces the standard Spiking Self-Attention (SSA) head with a Fourier-or-Wavelet (FW) head, utilizing fixed bases to reduce computational complexity.

By implementing these as "FW heads" within the Spikformer architecture, the model utilizes Fast Fourier Transform (FFT) algorithms to reduce computational complexity from $O(N^2)$ to $O(N \log N)$. Additionally, the authors introduced DWT-C, a hybrid approach that combines multiple wavelet bases with learnable coefficients, allowing the network to learn non-orthogonal basis functions as it deepens.

Experimental Results

The FWformer was evaluated on event-based video datasets (CIFAR10-DVS and DvsGesture) and static image datasets (CIFAR10/100). The results demonstrate that the FWformer outperformed the standard Spikformer across nearly every metric:

  • Accuracy: Improved Top-1 accuracy by 0.3%–1.5% on event-based datasets (reaching 81.2% on CIFAR10-DVS and 99.5% on DvsGesture).
  • Speed: Training was 9%–51% faster, and inference speed saw a massive jump of 19%–70%.
  • Efficiency: Theoretical energy consumption was reduced by 20%–25%, and GPU memory usage dropped by 4%–26%.

Conclusion

The FWformer demonstrates that in the context of sparse spikes, the "prior knowledge" embedded in Fourier or Wavelet bases provides a structured way to extract features without the massive parameter count of traditional attention heads. While fixed bases work exceptionally well for visual classification, the authors note that dynamic attention may still be necessary for more complex "semantic" tasks like Natural Language Processing (NLP). Future work will explore testing the FWformer on real-world neuromorphic hardware to further validate its energy-saving potential.

Summary updated

Abstract

Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer, whereby the Spiking Self-Attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this paper, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier Transform, Wavelet Transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies ($0.4\%$-$1.5\%$), higher running speed ($9\%$-$51\%$ for training and $19\%$-$70\%$ for inference), reduced theoretical energy consumption ($20\%$-$25\%$), and reduced GPU memory usage ($4\%$-$26\%$), compared to the standard spikformer. Our result indicates the continuous refinement of new Transformers, that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet Transform), is promising.
BibTeX
@inproceedings{wang2025fourier,
  author = {Qingyu Wang and Duzhen Zhang and Tielin Zhang and Bo Xu},
  title = {{Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification}},
  booktitle = {Frontiers in Neuroscience},
  year = {2025},
  doi = {10.3389/fnins.2024.1516868},
}
Figure