Abstract
Spiking Neural Networks (SNNs) offer a promising, energy-efficient paradigm for processing data from event cameras. However, they frequently suffer from a significant accuracy-latency trade-off, requiring many time steps to achieve high performance. This paper presents a novel training framework, Hybrid Step-wise Distillation (HSD), designed to achieve state-of-the-art accuracy at ultra-low latency (e.g., 5 time steps). By disentangling training and inference time steps and implementing a step-wise knowledge distillation technique, the proposed method stabilizes training and enhances feature representation in sparse data environments.
Introduction
Event cameras are bio-inspired sensors that record changes in light intensity asynchronously, producing a stream of "events" rather than traditional image frames. This sparse data format is ideally suited for Spiking Neural Networks (SNNs), which communicate via discrete binary spikes. While SNNs are theoretically more efficient than standard Artificial Neural Networks (ANNs) for this task, they face a critical bottleneck: latency.
To reach high accuracy, SNNs typically need to accumulate information over a long temporal window. When forced to operate at low latency (1–5 time steps), performance often drops sharply because the network lacks sufficient information to form reliable representations. This research addresses this "accuracy-latency trade-off" through a hybrid training and distillation strategy.
Methodology
The researchers propose a two-part strategy to bridge the gap between low latency and high performance: Hybrid Step-wise Distillation (HSD).
1. Hybrid Training Module
Traditionally, SNNs are trained using the same number of time steps as they will use during inference. HSD "disentangles" these two phases. During a pre-training phase, an ANN acts as a teacher, learning from a larger set of event frames ($T_1$) to capture rich spatial features. These weights are then converted into an SNN. In the subsequent fine-tuning phase, the SNN is optimized on a smaller subset of frames ($T_2$) to prepare it for low-latency inference. This allows the model to "learn" from high-density data while "acting" on low-density data.
2. Step-wise Knowledge Distillation (SKD)
Standard knowledge distillation typically involves a student model (SNN) matching the final average output of a teacher model (ANN). However, at low time steps, individual moments in an SNN can produce "abnormal" outputs due to the inherent sparsity of event data.
Step-wise Knowledge Distillation (SKD) forces the SNN to match the teacher's distribution at every single time step, rather than just the final output. This temporal supervision stabilizes the training process, effectively smoothing out temporal noise and making the network more robust to the outliers common in low-latency scenarios.
Experimental Results
The HSD framework was evaluated on three standard neuromorphic benchmarks: CIFAR10-DVS, N-Caltech101, and DVS-Gesture. The results demonstrate that HSD consistently outperforms existing state-of-the-art (SOTA) methods, particularly when latency is restricted to just 5 time steps.
Performance Benchmarks
- CIFAR10-DVS: The model achieved 81.10% accuracy at $T=5$, representing a 2.30% improvement over the previous SOTA (TET) and significantly outperforming other methods like SSNN (73.63%).
- N-Caltech101: The framework reached 80.20% accuracy at $T=5$, surpassing recent models such as TCJA and SLTT.
- DVS-Gesture: The method achieved a peak accuracy of 97.92%, demonstrating its ability to scale to complex, real-world temporal data.
Computational Efficiency
Beyond accuracy, the HSD method leverages Integrate-and-Fire (IF) neurons. Because these neurons rely on addition rather than multiplication and require fewer time steps to reach a decision, the resulting models are significantly more hardware-efficient than standard ANNs or high-latency SNNs.
Discussion and Insights
The success of HSD highlights two critical insights for the neuromorphic community. First, the "disentanglement" of training and inference budgets suggests that we should not limit a model's learning capacity by its eventual inference constraints. By providing the model with more information during training, we can bake superior feature representations into the weights.
Second, the ANN-SNN synergy is vital. While ANNs are superior at extracting spatial features from event frames, SNNs excel at temporal processing. Combining them via conversion and step-wise distillation leverages the strengths of both architectures.
Limitations and Future Work
While HSD provides a significant leap in low-latency performance, some limitations remain. On smaller datasets like DVS-Gesture, the improvement from the Hybrid Training module was less pronounced, as the model quickly saturated the available information. Future research may explore heterogeneous architectures—mixing different types of ANN and SNN layers—to further optimize the balance between power and precision. Additionally, extending this step-wise distillation to complex tasks like object detection and motion estimation remains a high-priority challenge.
Summary updated