Abstract
Spiking Neural Networks (SNNs) offer high energy efficiency for neuromorphic hardware but are notoriously difficult to train due to the non-differentiable nature of discrete spikes. This paper introduces S-SWIM, a gradient-free, data-driven algorithm that leverages Random Feature Methods (RFMs) to train SNNs. By constructing hidden layer weights to maximize latent space separation and solving the output layer via linear regression, S-SWIM achieves training speedups of 10x to 1000x compared to standard backpropagation while maintaining competitive accuracy, particularly in long-term time-series forecasting.
Introduction
The primary appeal of Spiking Neural Networks (SNNs) lies in their sparse, event-driven processing, which mimics biological neural activity and enables extreme energy efficiency. However, the discrete "step function" nature of a spike makes standard gradient-based optimization (backpropagation) problematic. Current workarounds, such as surrogate gradients, are computationally intensive and often introduce systematic biases.
To address these challenges, this research proposes a shift away from iterative gradient descent toward a data-driven random feature approach. The resulting algorithm, S-SWIM (Spiking-SWIM), enables end-to-end training without calculating gradients through the spike function.
The S-SWIM Framework
The S-SWIM architecture is based on the Spike Response Model (SRM). Unlike standard deep learning where all weights are learned iteratively, S-SWIM adopts the Random Feature Method paradigm: hidden layer weights are determined through a structured sampling process, and only the final output layer is trained using linear regression.
Data-Driven Weight Sampling
Purely random weights often require excessively large networks to achieve high performance. S-SWIM improves upon this by using a data-driven sampling algorithm. Instead of random initialization, it identifies pairs of input samples with significantly different target outputs. It then mathematically constructs hidden layer weights to maximize the separation of these specific inputs in the network's latent space.
Temporal and Voltage Normalization
SNNs are inherently temporal. S-SWIM incorporates several mechanisms to handle time-series data:
- Temporal Parameter Tuning: The algorithm uses correlation analysis to determine optimal delays for the output layer and a fast search method for tuning kernel supports.
- Voltage Normalization: To prevent "silent" neurons or runaway firing rates, weights and biases are scaled based on the statistical mean and variance of the input data, ensuring stable information propagation.
Experimental Results
The authors evaluated S-SWIM across diverse tasks, including time-series forecasting (traffic, solar energy, electricity) and classification (MNIST, Fashion-MNIST, and the neuromorphic SHD dataset).
Massive Training Speedups
The most striking result is the efficiency gain. S-SWIM trains networks one to three orders of magnitude faster than standard stochastic gradient descent (SGD) with surrogate gradients. Training times that previously took thousands of seconds were reduced to tens of seconds.
Superior Long-Term Forecasting
In time-series tasks, S-SWIM frequently outperformed standard SGD, particularly at longer prediction horizons (e.g., 48 or 96 steps ahead). This is attributed to the fact that S-SWIM bypasses the "credit assignment problem"—the difficulty gradient descent has in tracking errors over long temporal sequences.
Limitations and Conclusion
While S-SWIM excels in shallow architectures and forecasting, it faces challenges when scaling to deeper networks, a common limitation of Random Feature Methods. Additionally, some temporal parameters still rely on heuristics, and the current implementation requires real-valued target voltages for the output layer.
Despite these limitations, S-SWIM represents a significant step forward in making SNNs practical. It provides a powerful alternative to gradient-based training and can even serve as a superior initialization method for practitioners who still wish to use fine-tuning via gradient descent.
Summary updated