Abstract
Spiking neural networks (SNNs) present an increasingly popular alternative to artificial neural networks (ANNs), due to their energy and time efficiency when deployed on neuromorphic hardware. However, due to their discrete and highly non-differentiable nature, training SNNs is a challenging task and remains an active area of research. Some of the most prominent ways to train SNNs are based on ANN-to-SNN conversion where an SNN model is initialized with parameters from the corresponding, pre-trained ANN model. SNN models trained through ANN-to-SNN conversion or hybrid training show state of the art performance among SNNs on many machine learning tasks, comparable to those of ANNs. However, the top performing models need high latency or tailored ANNs to perform well, and in general are not using the full information available from ANNs. In this work, we propose novel method to initialize SNN’s thresholds and initial membrane potential after ANN-to-SNN conversion, using distributions of ANN’s activation values. We provide a theoretical framework for feature distribution-based conversion error, providing theoretical results on optimal membrane initialization and thresholds which minimize this error, as well as a practical algorithm for finding these optimal values. We test our method, both as a stand-alone ANN-to-SNN conversion and in combination with other methods, and show state of the art results on high-dimensional datasets such as CIFAR10, CIFAR100 and ImageNet and various architectures. Our code is available at https://github.com/srinuvaasu/data_driven_init.
Introduction
Spiking Neural Networks (SNNs) are prized for their energy efficiency, communicating via discrete binary spikes rather than continuous floating-point values. This makes them ideal for deployment on neuromorphic hardware. However, training SNNs is difficult due to the non-differentiable nature of spikes. A common workaround is ANN-to-SNN conversion, where a standard ANN is trained first and then converted into an SNN.
The primary challenge in conversion is the "latency gap": SNNs typically require many time steps to accumulate enough spikes to match the precision of the original ANN. Previous methods often relied on simple heuristics, such as setting thresholds to maximum activation values, which leads to high conversion errors. This paper introduces a data-driven method to initialize SNN thresholds and initial membrane potentials based on the actual activation distributions of a pre-trained ANN, significantly reducing latency.
Methodology
The authors propose a theoretical framework that treats conversion error as a function of the probability distribution of ANN activations. Their core contribution involves optimizing two key parameters:
- Threshold ($V_{th}$): The voltage at which a neuron fires.
- Initial Membrane Potential ($v[0]$): The starting charge of a neuron before receiving input.
By analyzing layer-wise conversion error, they derive optimal values for these parameters. A key theoretical finding is that for a latency of $T=1$, the optimal initial potential is exactly half the threshold ($v[0] = \frac{1}{2}V_{th}$). This initialization acts as a bias shift that centers quantization error, allowing the SNN to represent continuous values accurately from the very first time step.
To implement this, the authors provide two practical algorithms: a linear-time $O(N)$ algorithm for the $T=1$ case and an $O(N^2 \log N)$ algorithm for higher latencies ($T > 1$). These algorithms use a representative set of ANN activations to calibrate the SNN layers.
Experimental Results
The method was evaluated on standard computer vision benchmarks including CIFAR-10, CIFAR-100, and ImageNet across various architectures like VGG16 and ResNet.
Performance and Efficiency
The proposed method achieves state-of-the-art (SOTA) results, particularly in ultra-low latency scenarios. On CIFAR-10 with ResNet18, the method achieved 93.00% accuracy at $T=1$. On ImageNet with VGG16, it reached 69.28% accuracy at $T=2$, significantly outperforming existing baselines that require 32 or more time steps.
Beyond accuracy, the energy efficiency gains are substantial. For VGG16 on CIFAR100, the SNN at $T=1$ consumed only 0.0064 mJ, compared to 7.85 mJ for the original ANN—a massive reduction in power consumption.
Conclusion
This research demonstrates that leveraging the statistical distribution of ANN activations allows for near-lossless ANN-to-SNN conversion at extremely low latencies. By optimizing thresholds and initial membrane potentials, SNNs can achieve high accuracy while maintaining the energy efficiency benefits of neuromorphic hardware. Future work may explore channel-wise thresholding and extending the framework to non-ReLU activation functions.
Summary updated