Random Feature Spiking Neural Networks

Maximilian Gollwitzer, Felix DietrichSubmission to ICLR 2026 · Rejected

TL;DR

Training Spiking Neural Networks (SNNs) with gradient descent is computationally punishing and hindered by non-differentiable spike thresholds. S-SWIM extends data-driven Random Feature Methods (RFMs) to Spike Response Model SNNs, completely bypassing surrogate-gradient approximations. By sampling hidden layer weights from data-informed distributions and solving output weights analytically, S-SWIM trains SNNs one to three orders of magnitude faster than gradient descent while achieving competitive accuracy on time-series forecasting.

What Problem Does This Paper Address?

SNNs process information through sparse, discrete events (spikes), making them attractive for low-power neuromorphic hardware. However, standard backpropagation fails because the thresholding function that triggers a spike has an undefined or zero derivative almost everywhere.

The prevailing solution—surrogate-gradient backpropagation through time (e.g., SLAYER, STBP)—has severe drawbacks:

  • It introduces systematic gradient bias.
  • Memory and computational costs scale heavily with the number of simulated time steps.
  • Models suffer from the credit assignment problem over long horizons and are sensitive to hyperparameter tuning.

While Random Feature Methods (RFMs) like Extreme Learning Machines (ELMs) allow rapid, closed-form network training, classical RFMs choose hidden parameters completely at random (data-agnostically), requiring enormous networks to perform well. S-SWIM bridges this gap by adapting the data-driven SWIM algorithm to the spatio-temporal dynamics of spiking neurons.

Methods

S-SWIM formulates feedforward SNNs using the Spike Response Model (SRM), parameterizing input convolution kernels (Spike Response Kernels, SRK) and refractory kernels (RfK). Training is structured without any backpropagation:

  1. Temporal Parameters Initialization: Hidden layer delays ($\tau$) and kernel support widths ($\sigma$) are distributed deterministically across multiple timescales to capture diverse temporal dependencies.
  2. Data-Driven Sampling Distribution: Rather than uniform weight sampling, S-SWIM samples pairs of input samples with probability proportional to the ratio of their target dissimilarity to their input dissimilarity, focusing capacity on regions with steep function variations.
  3. Weight Construction via Eigenproblems: For each neuron, candidate weight directions ($W_{\text{dist}}$ or $W_{\text{dot}}$) are determined by solving a small eigenvalue problem that maximizes output voltage separation or minimizes cross-correlation between the sampled input pair.
  4. Voltage Normalization & Silence Correction: Weights and biases are scaled so membrane potential fluctuations stay within an active firing range, with a "silence correction" adjustment guaranteeing every neuron fires at least once.
  5. Output Delay & Support Estimation: Output layer delays are estimated using linear cross-correlation between hidden spike trains and target time series. Kernel supports are chosen via a fast residual-norm search over thin QR decompositions of the design matrix.
  6. Closed-Form Output Weights: Output weights and biases are solved directly using regularized normal equations accumulated in mini-batches.

Datasets

The authors evaluate S-SWIM on two main problem classes (no new benchmarks were introduced):

  • Time-Series Forecasting:
  • Metr-la and Pems-bay (short-term traffic observation, lookback = 12).
  • Solar-energy and Electricity (long-term observation, lookback = 168).
  • Classification:
  • MNIST and Fashion-MNIST (static image datasets converted to Poisson spike trains).
  • Spiking Heidelberg Digits (SHD) (neuromorphic speech dataset, evaluated with 70 merged channels).

Key Insights and Ideas

  • Separation Without Gradients: Rather than guessing gradient shapes across a discontinuous threshold, S-SWIM constructs hidden weights that explicitly separate voltage traces for inputs whose target outputs differ.
  • Correlation Replaces Temporal Optimization: Cross-correlating hidden spike trains with output targets reveals optimal delays directly from the timing of kernel peaks, removing the need for iterative delay gradient updates.
  • Bypassing the Temporal Credit Assignment Bottleneck: Because gradient backpropagation struggles over long time horizons, S-SWIM often matches or outperforms gradient-trained SNNs on longer forecast windows ($H = 48, 96$).

Key Results

  • Massive Speedups: S-SWIM achieves $10\times$ to $1000\times$ faster training compared to Adam/SLAYER. For instance, on the Solar dataset with horizon $H=96$, S-SWIM finishes in ~222 seconds versus ~49,800 seconds for SGD.
  • Forecasting Performance: On Solar and Electricity benchmarks, standalone S-SWIM matches or outperforms SGD baselines and is competitive with Spikformer/Transformer baselines in terms of Relative Squared Error (RSE). For Electricity ($H=96$), S-SWIM scores 0.370 RSE, outperforming the pure SGD baseline (0.570–0.609).
  • Initialization for Fine-Tuning: Using S-SWIM to initialize weights before brief SGD fine-tuning converges faster and achieves the best overall scores across multiple long-horizon benchmarks (e.g., Solar $H=96$ drops to 0.267 RSE).
  • Data-Driven vs. Random Ablation: The data-driven $W_{\text{dot}}$ criterion consistently and significantly outperforms standard Gaussian random weight initialization across neuron counts.

Limitations and Future Work

  • Depth Degradation: S-SWIM performance drops when extending from single hidden layers ($L=1$) to deeper architectures ($L=2$), a recognized challenge in Random Feature Methods.
  • Short-Horizon Underperformance: On very short forecast horizons ($H=6$), S-SWIM lags behind SGD, likely due to aggregating output delays to simplify kernel support searches.
  • Heuristic Parameter Selection: Delays and kernel supports in internal hidden layers are still chosen via deterministic heuristics rather than learned directly from data spectra.
  • Spike-Valued Outputs: For pure spike-in/spike-out tasks (e.g., neuromorphic classification), S-SWIM relies on hand-crafted surrogate target voltages, leading to lower classification accuracy compared to real-valued voltage readouts.

Summary updated

Abstract

Spiking Neural Networks (\textit{SNN}s) as Machine Learning (\textit{ML}) models have recently received a lot of attention as a potentially more energy-efficient alternative to conventional Artificial Neural Networks. The non-differentiability and sparsity of the spiking mechanism can make these models very difficult to train with algorithms based on propagating gradients through the spiking non-linearity. We address this problem by adapting the paradigm of Random Feature Methods (\textit{RFM}s) from Artificial Neural Networks (\textit{ANN}s) to Spike Response Model (\textit{SRM}) \textit{SNN}s. This approach allows training of \textit{SNN}s without approximation of the spike function gradient. Concretely, we propose a novel data-driven, fast, high-performance, and interpretable algorithm for end-to-end training of \textit{SNN}s inspired by the \textit{SWIM} algorithm for \textit{RFM}-\textit{ANN}s, which we coin \textit{S-SWIM}. We provide a thorough theoretical discussion and supplementary numerical experiments showing that \textit{S-SWIM} can reach high accuracies on time series forecasting as a standalone strategy and serve as an effective initialisation strategy before gradient-based training. Additional ablation studies show that our proposed method performs better than random sampling of network weights.
BibTeX
@misc{gollwitzer2025random,
  author = {Maximilian Gollwitzer and Felix Dietrich},
  title = {{Random Feature Spiking Neural Networks}},
  year = {2025},
  eprint = {2510.01012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01012},
}
Figure