Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers

Yichen Xiao, Shuai Wang, Dehao Zhang, Wenjie Wei, Yimeng Shan, Xiaoli Liu, Yulin Jiang, Malu Zhang

Abstract

Spiking Neural Networks (SNNs) are energy-efficient alternatives to traditional Artificial Neural Networks (ANNs), yet they struggle to match performance in Transformer architectures. This research identifies a critical limitation in the standard Spiking Self-Attention (SSA) mechanism: the dot-product similarity calculation fails to account for "matching silences" (0-0 pairs), leading to substantial information loss. We introduce α-XNOR similarity, a novel metric that rewards both spike and non-spike matches with tunable weights. By incorporating this into an α-SSA module, we significantly enhance the information capacity of SNNs, achieving state-of-the-art performance across multiple benchmarks, including an 80.62% top-1 accuracy on ImageNet-1K.

The Limitation of Dot-Product in Spiking Networks

In traditional Transformers, self-attention relies on the dot product between Query ($Q$) and Key ($K$) vectors. In the binary domain of SNNs, where signals consist of spikes (1) and silences (0), this operation becomes problematic. Because $0 \times 0 = 0$ and $1 \times 0 = 0$, the dot product cannot distinguish between a perfect match of silences and a complete mismatch.

Figure 2: Illustration of the dot-product "blind spot" in SNNs. Note how q3 and k3 have 8 matching values (all zeros), yet result in a Similarity Score (SS) of 0.

Given that spike trains are inherently sparse, the majority of potential correlations reside in these "matching silences." Ignoring them results in low information entropy and "unreasonable" similarity scores that fail to capture the true relationship between tokens.

The α-XNOR Similarity Measure

To address this, we propose the α-XNOR similarity measure. Unlike the dot product, α-XNOR explicitly recognizes the correlation of silence:

  1. Spike Match (1, 1): Assigned a score of 1. These are high-information events.
  2. Non-Spike Match (0, 0): Assigned a score of α ($0 < α < 1$). This acknowledges that shared silence indicates correlation, though it is weighted less than active spikes.
  3. Mismatch (1, 0 or 0, 1): Assigned a score of 0.
Figure 6: The α-XNOR similarity calculation process. By adjusting α, the network can tune the importance of matching silences relative to matching spikes.

This mechanism allows the attention map to become more expressive. Theoretical analysis confirms that α-XNOR similarity leads to higher information entropy, producing attention maps that are more focused on salient visual features.

Implementation and Training

The α-XNOR operation is integrated into a Spiking Self-Attention (α-SSA) module. To overcome the non-differentiable nature of the XNOR logic, we utilize custom gradients for backpropagation. Additionally, we implement a distribution balancing step—a linear transformation ($k \cdot \text{Score} + b$)—to normalize attention scores before they reach subsequent layers, ensuring stable training and optimal neuron firing rates.

Experimental Results

The proposed α-SSA module was evaluated against leading spiking Transformer baselines (e.g., Spikformer, Spike-driven Transformer) on several datasets:

  • ImageNet-1K: Achieved a landmark 80.62% accuracy, demonstrating the scalability of the approach.
  • CIFAR-100: Reached 80.22% accuracy, a 2.01% improvement over the standard Spikformer.
  • Neuromorphic Data (CIFAR10-DVS): Set a new state-of-the-art with 82.22% accuracy.

Ablation studies indicate that the "Goldilocks Zone" for the hyperparameter α lies between 0.2 and 0.4. This range provides enough weight to matching silences to boost information capacity without overshadowing the importance of sparse spikes.

Conclusion

By rethinking the fundamental similarity operation in Spiking Transformers, we demonstrate that "silence" in SNNs is not merely an absence of data but a valuable source of correlation. The α-XNOR similarity measure effectively bridges the performance gap between SNNs and ANNs while maintaining the energy efficiency inherent to neuromorphic computing. Future work will focus on automated α tuning and specialized hardware kernels to further optimize these brain-inspired architectures.

Summary updated

Abstract

Transformers significantly raise the performance limits across various tasks, spurring research into integrating them into spiking neural networks. However, a notable performance gap remains between existing spiking Transformers and their artificial neural network counterparts. Here, we first analyze the cause of this gap and attribute it to the dot product's ineffectiveness in measuring similarity between spiking queries and keys, due to numerous non-spiking events. To address this, we propose a novel a-XNOR similarity measure tailored for spike trains. It redefines the correlation between non-spike pairs as a specific value a, effectively overcoming the limitations of dot-product similarity. Furthermore, considering the sparse nature of spike trains where spikes carry more information than non-spikes, the a-XNOR similarity correspondingly highlights the distinct importance of spikes over non-spikes. Extensive experiments demonstrate that a-XNOR similarity significantly improves performance across different spiking Transformer architectures on various static and neuromorphic datasets, further revealing the potential of spiking Transformers.
BibTeX
@inproceedings{xiao2025rethinking,
  author = {Yichen Xiao and Shuai Wang and Dehao Zhang and Wenjie Wei and Yimeng Shan and Xiaoli Liu and Yulin Jiang and Malu Zhang},
  title = {{Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00512},
}
Figure