Abstract
Spiking Neural Networks (SNNs) are energy-efficient alternatives to traditional Artificial Neural Networks (ANNs), yet they struggle to match performance in Transformer architectures. This research identifies a critical limitation in the standard Spiking Self-Attention (SSA) mechanism: the dot-product similarity calculation fails to account for "matching silences" (0-0 pairs), leading to substantial information loss. We introduce α-XNOR similarity, a novel metric that rewards both spike and non-spike matches with tunable weights. By incorporating this into an α-SSA module, we significantly enhance the information capacity of SNNs, achieving state-of-the-art performance across multiple benchmarks, including an 80.62% top-1 accuracy on ImageNet-1K.
The Limitation of Dot-Product in Spiking Networks
In traditional Transformers, self-attention relies on the dot product between Query ($Q$) and Key ($K$) vectors. In the binary domain of SNNs, where signals consist of spikes (1) and silences (0), this operation becomes problematic. Because $0 \times 0 = 0$ and $1 \times 0 = 0$, the dot product cannot distinguish between a perfect match of silences and a complete mismatch.
Given that spike trains are inherently sparse, the majority of potential correlations reside in these "matching silences." Ignoring them results in low information entropy and "unreasonable" similarity scores that fail to capture the true relationship between tokens.
The α-XNOR Similarity Measure
To address this, we propose the α-XNOR similarity measure. Unlike the dot product, α-XNOR explicitly recognizes the correlation of silence:
- Spike Match (1, 1): Assigned a score of 1. These are high-information events.
- Non-Spike Match (0, 0): Assigned a score of α ($0 < α < 1$). This acknowledges that shared silence indicates correlation, though it is weighted less than active spikes.
- Mismatch (1, 0 or 0, 1): Assigned a score of 0.
This mechanism allows the attention map to become more expressive. Theoretical analysis confirms that α-XNOR similarity leads to higher information entropy, producing attention maps that are more focused on salient visual features.
Implementation and Training
The α-XNOR operation is integrated into a Spiking Self-Attention (α-SSA) module. To overcome the non-differentiable nature of the XNOR logic, we utilize custom gradients for backpropagation. Additionally, we implement a distribution balancing step—a linear transformation ($k \cdot \text{Score} + b$)—to normalize attention scores before they reach subsequent layers, ensuring stable training and optimal neuron firing rates.
Experimental Results
The proposed α-SSA module was evaluated against leading spiking Transformer baselines (e.g., Spikformer, Spike-driven Transformer) on several datasets:
- ImageNet-1K: Achieved a landmark 80.62% accuracy, demonstrating the scalability of the approach.
- CIFAR-100: Reached 80.22% accuracy, a 2.01% improvement over the standard Spikformer.
- Neuromorphic Data (CIFAR10-DVS): Set a new state-of-the-art with 82.22% accuracy.
Ablation studies indicate that the "Goldilocks Zone" for the hyperparameter α lies between 0.2 and 0.4. This range provides enough weight to matching silences to boost information capacity without overshadowing the importance of sparse spikes.
Conclusion
By rethinking the fundamental similarity operation in Spiking Transformers, we demonstrate that "silence" in SNNs is not merely an absence of data but a valuable source of correlation. The α-XNOR similarity measure effectively bridges the performance gap between SNNs and ANNs while maintaining the energy efficiency inherent to neuromorphic computing. Future work will focus on automated α tuning and specialized hardware kernels to further optimize these brain-inspired architectures.
Summary updated