Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization

Lucas Deckers, Benjamin Vandersmissen, Ing Jyh Tsang, Werner Van Leekwijck, Steven LatréarXiv 2024 · DOI · arXiv

Introduction

Spiking Neural Networks (SNNs) offer a biologically inspired, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) by utilizing sparse, asynchronous spikes for communication. Despite their potential for low-power hardware, SNNs face significant challenges, including a performance gap compared to ANNs and high sensitivity to weight initialization. Furthermore, achieving extreme weight quantization (e.g., ternary weights) without sacrificing accuracy remains difficult.

This paper introduces Twin Network Augmentation (TNA), a training strategy designed to improve SNN generalization and enable high-performance ternary quantization.

Methodology: Twin Network Augmentation

The TNA approach involves co-training two identical SNNs—a "base" network and a "twin" network—starting from different random weight initializations.

Co-training and Logit Matching

Both networks are trained simultaneously from scratch. The training objective incorporates a Logit Matching Loss, which uses Mean Squared Error (MSE) to penalize differences between the output logits of the two networks. This forces the "siblings" to find a consensus, effectively regularizing the learning process by leveraging the diverse "views" provided by different initializations.

Overview of the TNA training and inference workflow. Training utilizes both base and twin networks with a combined loss, while inference only requires the base network.

Delayed Ternary Quantization

To optimize for hardware deployment, the base network is compressed to ternary weights (restricted to -1, 0, and 1). This process occurs after an initial full-precision training phase. During quantization fine-tuning, the twin network remains in full precision, providing a stable reference that helps the base network maintain accuracy despite the reduced precision. Upon completion, the twin network is discarded, leaving a lightweight, efficient model for deployment.

Experimental Results

TNA was evaluated on several benchmarks, including CIFAR-10, CIFAR-100, Fashion-MNIST, and the neuromorphic CIFAR-10-DVS dataset.

Accuracy and Quantization Performance

The results demonstrate that TNA consistently outperforms standard training and traditional Knowledge Distillation (KD). On CIFAR-100, TNA provided a 2.4% accuracy boost over the baseline. Remarkably, a ternary SNN trained with TNA on CIFAR-100 achieved 72.03% accuracy, outperforming some full-precision models.

Accuracy comparison on CIFAR-100. The right panel highlights that TNA (Twin network augment.) achieves superior performance compared to single SNN baselines and standard Knowledge Distillation.

State-of-the-Art Benchmarks

Using the CIFARNet architecture, TNA established new performance records for its class across CIFAR-10, CIFAR-100, and CIFAR-10-DVS. On Fashion-MNIST, the model reached 95.23% accuracy, surpassing several larger and more complex architectures.

Conclusion and Future Work

Twin Network Augmentation provides a robust framework for training high-performance SNNs while facilitating efficient weight quantization. By treating initialization as a source of diversity and enforcing consensus, TNA bridges the gap between SNNs and traditional ANNs. Future research will explore scaling TNA to deeper architectures like ResNets and investigating heterogeneous co-training strategies.

Summary updated

Abstract

The proliferation of Artificial Neural Networks (ANNs) has led to increased energy consumption, raising concerns about their sustainability. Spiking Neural Networks (SNNs), which are inspired by biological neural systems and operate using sparse, event-driven spikes to communicate information between neurons, offer a potential solution due to their lower energy requirements. An alternative technique for reducing a neural network's footprint is quantization, which compresses weight representations to decrease memory usage and energy consumption. In this study, we present Twin Network Augmentation (TNA), a novel training framework aimed at improving the performance of SNNs while also facilitating an enhanced compression through low-precision quantization of weights. TNA involves co-training an SNN with a twin network, optimizing both networks to minimize their cross-entropy losses and the mean squared error between their output logits. We demonstrate that TNA significantly enhances classification performance across various vision datasets and in addition is particularly effective when applied when reducing SNNs to ternary weight precision. Notably, during inference , only the ternary SNN is retained, significantly reducing the network in number of neurons, connectivity and weight size representation. Our results show that TNA outperforms traditional knowledge distillation methods and achieves state-of-the-art performance for the evaluated network architecture on benchmark datasets, including CIFAR-10, CIFAR-100, and CIFAR-10-DVS. This paper underscores the effectiveness of TNA in bridging the performance gap between SNNs and ANNs and suggests further exploration into the application of TNA in different network architectures and datasets.
BibTeX
@misc{deckers2024twin,
  author = {Lucas Deckers and Benjamin Vandersmissen and Ing Jyh Tsang and Werner Van Leekwijck and Steven Latré},
  title = {{Twin Network Augmentation: A Novel Training Strategy for Improved Spiking Neural Networks and Efficient Weight Quantization}},
  year = {2024},
  eprint = {2409.15849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15849},
}
Figure