TL;DR
Training Spiking Neural Networks (SNNs) with gradient descent is computationally punishing and hindered by non-differentiable spike thresholds. S-SWIM extends data-driven Random Feature Methods (RFMs) to Spike Response Model SNNs, completely bypassing surrogate-gradient approximations. By sampling hidden layer weights from data-informed distributions and solving output weights analytically, S-SWIM trains SNNs one to three orders of magnitude faster than gradient descent while achieving competitive accuracy on time-series forecasting.
What Problem Does This Paper Address?
SNNs process information through sparse, discrete events (spikes), making them attractive for low-power neuromorphic hardware. However, standard backpropagation fails because the thresholding function that triggers a spike has an undefined or zero derivative almost everywhere.
The prevailing solution—surrogate-gradient backpropagation through time (e.g., SLAYER, STBP)—has severe drawbacks:
- It introduces systematic gradient bias.
- Memory and computational costs scale heavily with the number of simulated time steps.
- Models suffer from the credit assignment problem over long horizons and are sensitive to hyperparameter tuning.
While Random Feature Methods (RFMs) like Extreme Learning Machines (ELMs) allow rapid, closed-form network training, classical RFMs choose hidden parameters completely at random (data-agnostically), requiring enormous networks to perform well. S-SWIM bridges this gap by adapting the data-driven SWIM algorithm to the spatio-temporal dynamics of spiking neurons.
Methods
S-SWIM formulates feedforward SNNs using the Spike Response Model (SRM), parameterizing input convolution kernels (Spike Response Kernels, SRK) and refractory kernels (RfK). Training is structured without any backpropagation:
- Temporal Parameters Initialization: Hidden layer delays ($\tau$) and kernel support widths ($\sigma$) are distributed deterministically across multiple timescales to capture diverse temporal dependencies.
- Data-Driven Sampling Distribution: Rather than uniform weight sampling, S-SWIM samples pairs of input samples with probability proportional to the ratio of their target dissimilarity to their input dissimilarity, focusing capacity on regions with steep function variations.
- Weight Construction via Eigenproblems: For each neuron, candidate weight directions ($W_{\text{dist}}$ or $W_{\text{dot}}$) are determined by solving a small eigenvalue problem that maximizes output voltage separation or minimizes cross-correlation between the sampled input pair.
- Voltage Normalization & Silence Correction: Weights and biases are scaled so membrane potential fluctuations stay within an active firing range, with a "silence correction" adjustment guaranteeing every neuron fires at least once.
- Output Delay & Support Estimation: Output layer delays are estimated using linear cross-correlation between hidden spike trains and target time series. Kernel supports are chosen via a fast residual-norm search over thin QR decompositions of the design matrix.
- Closed-Form Output Weights: Output weights and biases are solved directly using regularized normal equations accumulated in mini-batches.
Datasets
The authors evaluate S-SWIM on two main problem classes (no new benchmarks were introduced):
- Time-Series Forecasting:
- Metr-la and Pems-bay (short-term traffic observation, lookback = 12).
- Solar-energy and Electricity (long-term observation, lookback = 168).
- Classification:
- MNIST and Fashion-MNIST (static image datasets converted to Poisson spike trains).
- Spiking Heidelberg Digits (SHD) (neuromorphic speech dataset, evaluated with 70 merged channels).
Key Insights and Ideas
- Separation Without Gradients: Rather than guessing gradient shapes across a discontinuous threshold, S-SWIM constructs hidden weights that explicitly separate voltage traces for inputs whose target outputs differ.
- Correlation Replaces Temporal Optimization: Cross-correlating hidden spike trains with output targets reveals optimal delays directly from the timing of kernel peaks, removing the need for iterative delay gradient updates.
- Bypassing the Temporal Credit Assignment Bottleneck: Because gradient backpropagation struggles over long time horizons, S-SWIM often matches or outperforms gradient-trained SNNs on longer forecast windows ($H = 48, 96$).
Key Results
- Massive Speedups: S-SWIM achieves $10\times$ to $1000\times$ faster training compared to Adam/SLAYER. For instance, on the Solar dataset with horizon $H=96$, S-SWIM finishes in ~222 seconds versus ~49,800 seconds for SGD.
- Forecasting Performance: On Solar and Electricity benchmarks, standalone S-SWIM matches or outperforms SGD baselines and is competitive with Spikformer/Transformer baselines in terms of Relative Squared Error (RSE). For Electricity ($H=96$), S-SWIM scores 0.370 RSE, outperforming the pure SGD baseline (0.570–0.609).
- Initialization for Fine-Tuning: Using S-SWIM to initialize weights before brief SGD fine-tuning converges faster and achieves the best overall scores across multiple long-horizon benchmarks (e.g., Solar $H=96$ drops to 0.267 RSE).
- Data-Driven vs. Random Ablation: The data-driven $W_{\text{dot}}$ criterion consistently and significantly outperforms standard Gaussian random weight initialization across neuron counts.
Limitations and Future Work
- Depth Degradation: S-SWIM performance drops when extending from single hidden layers ($L=1$) to deeper architectures ($L=2$), a recognized challenge in Random Feature Methods.
- Short-Horizon Underperformance: On very short forecast horizons ($H=6$), S-SWIM lags behind SGD, likely due to aggregating output delays to simplify kernel support searches.
- Heuristic Parameter Selection: Delays and kernel supports in internal hidden layers are still chosen via deterministic heuristics rather than learned directly from data spectra.
- Spike-Valued Outputs: For pure spike-in/spike-out tasks (e.g., neuromorphic classification), S-SWIM relies on hand-crafted surrogate target voltages, leading to lower classification accuracy compared to real-valued voltage readouts.
Summary updated