TL;DR
This paper tackles a key challenge in training Spiking Neural Networks (SNNs): inconsistent output distributions across different timesteps, which hinders performance and necessitates long simulation times. They introduce a novel "Enhancing Temporal Consistency" (ETC) loss that encourages stable, consistent outputs, leading to state-of-the-art results on neuromorphic datasets and competitive performance on static ones, even with significantly reduced simulation timesteps.
What Problem Does This Paper Address?
Spiking Neural Networks (SNNs) are bio-inspired computational models that process information using discrete "spikes," offering potential for high energy efficiency and integration with neuromorphic hardware. However, training these networks, especially deep ones, has been historically difficult due to the non-differentiable nature of spikes. While backpropagation with surrogate gradients has enabled significant progress, a persistent issue remains: SNNs often require a large number of simulation timesteps to achieve good performance. This is because traditional training methods tend to average outputs across timesteps or rely on the final timestep's output, overlooking the fact that inputs at different moments can lead to varying output distributions. This temporal inconsistency can cause conflicting optimization signals during training and lead to misclassifications during inference, especially on spatiotemporal data like that from neuromorphic sensors. The authors aim to address this by improving the temporal consistency of SNN outputs.
Methods
The core of the proposed solution is the Enhancing Temporal Consistency (ETC) loss. This loss function is designed to minimize the divergence between the output probability distributions of an SNN at different timesteps. Specifically, they define the output probability distribution at each timestep $t$ as $P_t = \text{softmax}(V_t)$, where $V_t$ is the membrane potential of the output layer at timestep $t$. The ETC loss then uses the Kullback-Leibler (KL) divergence to measure the difference between the distributions of different timesteps, aiming to make them as similar as possible.
To achieve this, they formulate a new loss function that combines the standard cross-entropy loss ($L_{CE}$) with the proposed ETC loss ($L_{ETC}$): $L_{all} = L_{CE} + \lambda L_{ETC}$. The parameter $\lambda$ controls the influence of the ETC loss. Crucially, they avoid backpropagating gradients through all timesteps in the ETC loss to prevent model collapse, focusing on standardizing the output distribution. This approach effectively acts as a form of self-distillation, where the model's outputs at different timesteps provide "soft labels" for each other, guiding the network towards more stable and consistent temporal behavior.
Datasets
The paper evaluates their method on a range of datasets:
- Static Datasets:
- CIFAR-10
- CIFAR-100
- ImageNet
- Neuromorphic Datasets:
- DVS-CIFAR10 (event-based data)
- N-Caltech101 (event-based data)
These datasets are standard benchmarks for evaluating SNN performance, with the neuromorphic datasets being particularly relevant for testing the method's ability to handle rich spatiotemporal information.
Key Insights and Ideas
- Temporal Inconsistency is a Bottleneck: The paper clearly articulates how the inherent variability of SNN outputs across timesteps, especially for spatiotemporal data, leads to conflicting optimization signals and reduced performance. This is a crucial insight for understanding SNN training limitations.
- Self-Distillation for Temporal Stability: The ETC loss acts as a novel form of self-distillation, where the model learns from its own temporal variations. This is a clever way to leverage the rich temporal dynamics of SNNs without requiring external supervision for each timestep.
- Reduced Latency is Achievable: By enforcing temporal consistency, the model can achieve high accuracy with significantly fewer simulation timesteps. This is a critical practical advantage for deploying SNNs on resource-constrained edge devices.
- Neuromorphic Data Benefits Most: The method shows particularly strong gains on neuromorphic datasets, highlighting its effectiveness in handling the complex, time-varying nature of event-based data.
Key Results
- State-of-the-Art on Neuromorphic Datasets: The proposed ETC algorithm achieved state-of-the-art performance on DVS-CIFAR10 and N-Caltech101.
- Competitive Performance on Static Datasets: The method achieved comparable performance to other optimal SNN algorithms on CIFAR-10, CIFAR-100, and ImageNet.
- Significant Latency Reduction: For DVS-CIFAR10, with a timestep of $T=1$, their algorithm achieved superior performance, demonstrating the ability to achieve high accuracy with minimal simulation steps. For instance, on DVS-CIFAR10, their method achieved 63.1% accuracy at $T=1$, a significant improvement over the baseline's 43.3%.
- Improved Accuracy with Shorter Timesteps: On DVS-CIFAR10, their ETC algorithm achieved 63.1% accuracy at $T=1$, compared to the baseline's 43.3%. At $T=5$, their method surpassed the baseline's performance at $T=10$.
Limitations and Future Work
- Hyperparameter Sensitivity: The paper acknowledges that hyperparameters like $\tau$ (temperature parameter for smoothing output distribution) and $\lambda$ (weight for ETC loss) require careful tuning. If $\tau$ is too large, outputs become too smooth; if too small, they become overconfident. If $\lambda$ is too large or small, it can hinder effective loss guidance.
- Future Directions: The authors suggest drawing further inspiration from biological mechanisms for encoding and decoding information across different moments. They also hint at exploring more efficient coding methods that adaptively encode diverse inputs and decode outputs accurately, similar to biological neurons.
Summary updated