Introduction
Spiking Neural Networks (SNNs) offer a biologically inspired, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) by utilizing sparse, asynchronous spikes for communication. Despite their potential for low-power hardware, SNNs face significant challenges, including a performance gap compared to ANNs and high sensitivity to weight initialization. Furthermore, achieving extreme weight quantization (e.g., ternary weights) without sacrificing accuracy remains difficult.
This paper introduces Twin Network Augmentation (TNA), a training strategy designed to improve SNN generalization and enable high-performance ternary quantization.
Methodology: Twin Network Augmentation
The TNA approach involves co-training two identical SNNs—a "base" network and a "twin" network—starting from different random weight initializations.
Co-training and Logit Matching
Both networks are trained simultaneously from scratch. The training objective incorporates a Logit Matching Loss, which uses Mean Squared Error (MSE) to penalize differences between the output logits of the two networks. This forces the "siblings" to find a consensus, effectively regularizing the learning process by leveraging the diverse "views" provided by different initializations.
Delayed Ternary Quantization
To optimize for hardware deployment, the base network is compressed to ternary weights (restricted to -1, 0, and 1). This process occurs after an initial full-precision training phase. During quantization fine-tuning, the twin network remains in full precision, providing a stable reference that helps the base network maintain accuracy despite the reduced precision. Upon completion, the twin network is discarded, leaving a lightweight, efficient model for deployment.
Experimental Results
TNA was evaluated on several benchmarks, including CIFAR-10, CIFAR-100, Fashion-MNIST, and the neuromorphic CIFAR-10-DVS dataset.
Accuracy and Quantization Performance
The results demonstrate that TNA consistently outperforms standard training and traditional Knowledge Distillation (KD). On CIFAR-100, TNA provided a 2.4% accuracy boost over the baseline. Remarkably, a ternary SNN trained with TNA on CIFAR-100 achieved 72.03% accuracy, outperforming some full-precision models.
State-of-the-Art Benchmarks
Using the CIFARNet architecture, TNA established new performance records for its class across CIFAR-10, CIFAR-100, and CIFAR-10-DVS. On Fashion-MNIST, the model reached 95.23% accuracy, surpassing several larger and more complex architectures.
Conclusion and Future Work
Twin Network Augmentation provides a robust framework for training high-performance SNNs while facilitating efficient weight quantization. By treating initialization as a source of diversity and enforcing consensus, TNA bridges the gap between SNNs and traditional ANNs. Future research will explore scaling TNA to deeper architectures like ResNets and investigating heterogeneous co-training strategies.
Summary updated