Random Feature Spiking Neural Networks

Maximilian Gollwitzer, Felix Dietrich

Abstract

Spiking Neural Networks (SNNs) offer high energy efficiency for neuromorphic hardware but are notoriously difficult to train due to the non-differentiable nature of discrete spikes. This paper introduces S-SWIM, a gradient-free, data-driven algorithm that leverages Random Feature Methods (RFMs) to train SNNs. By constructing hidden layer weights to maximize latent space separation and solving the output layer via linear regression, S-SWIM achieves training speedups of 10x to 1000x compared to standard backpropagation while maintaining competitive accuracy, particularly in long-term time-series forecasting.

Introduction

The primary appeal of Spiking Neural Networks (SNNs) lies in their sparse, event-driven processing, which mimics biological neural activity and enables extreme energy efficiency. However, the discrete "step function" nature of a spike makes standard gradient-based optimization (backpropagation) problematic. Current workarounds, such as surrogate gradients, are computationally intensive and often introduce systematic biases.

To address these challenges, this research proposes a shift away from iterative gradient descent toward a data-driven random feature approach. The resulting algorithm, S-SWIM (Spiking-SWIM), enables end-to-end training without calculating gradients through the spike function.

The S-SWIM Framework

The S-SWIM architecture is based on the Spike Response Model (SRM). Unlike standard deep learning where all weights are learned iteratively, S-SWIM adopts the Random Feature Method paradigm: hidden layer weights are determined through a structured sampling process, and only the final output layer is trained using linear regression.

Overview of the S-SWIM architecture, SRM neuron model, and the data-driven weight sampling process.

Data-Driven Weight Sampling

Purely random weights often require excessively large networks to achieve high performance. S-SWIM improves upon this by using a data-driven sampling algorithm. Instead of random initialization, it identifies pairs of input samples with significantly different target outputs. It then mathematically constructs hidden layer weights to maximize the separation of these specific inputs in the network's latent space.

Temporal and Voltage Normalization

SNNs are inherently temporal. S-SWIM incorporates several mechanisms to handle time-series data:

  • Temporal Parameter Tuning: The algorithm uses correlation analysis to determine optimal delays for the output layer and a fast search method for tuning kernel supports.
  • Voltage Normalization: To prevent "silent" neurons or runaway firing rates, weights and biases are scaled based on the statistical mean and variance of the input data, ensuring stable information propagation.

Experimental Results

The authors evaluated S-SWIM across diverse tasks, including time-series forecasting (traffic, solar energy, electricity) and classification (MNIST, Fashion-MNIST, and the neuromorphic SHD dataset).

Massive Training Speedups

The most striking result is the efficiency gain. S-SWIM trains networks one to three orders of magnitude faster than standard stochastic gradient descent (SGD) with surrogate gradients. Training times that previously took thousands of seconds were reduced to tens of seconds.

Superior Long-Term Forecasting

In time-series tasks, S-SWIM frequently outperformed standard SGD, particularly at longer prediction horizons (e.g., 48 or 96 steps ahead). This is attributed to the fact that S-SWIM bypasses the "credit assignment problem"—the difficulty gradient descent has in tracking errors over long temporal sequences.

Limitations and Conclusion

While S-SWIM excels in shallow architectures and forecasting, it faces challenges when scaling to deeper networks, a common limitation of Random Feature Methods. Additionally, some temporal parameters still rely on heuristics, and the current implementation requires real-valued target voltages for the output layer.

Despite these limitations, S-SWIM represents a significant step forward in making SNNs practical. It provides a powerful alternative to gradient-based training and can even serve as a superior initialization method for practitioners who still wish to use fine-tuning via gradient descent.

Summary updated

Abstract

Spiking Neural Networks (\textit{SNN}s) as Machine Learning (\textit{ML}) models have recently received a lot of attention as a potentially more energy-efficient alternative to conventional Artificial Neural Networks. The non-differentiability and sparsity of the spiking mechanism can make these models very difficult to train with algorithms based on propagating gradients through the spiking non-linearity. We address this problem by adapting the paradigm of Random Feature Methods (\textit{RFM}s) from Artificial Neural Networks (\textit{ANN}s) to Spike Response Model (\textit{SRM}) \textit{SNN}s. This approach allows training of \textit{SNN}s without approximation of the spike function gradient. Concretely, we propose a novel data-driven, fast, high-performance, and interpretable algorithm for end-to-end training of \textit{SNN}s inspired by the \textit{SWIM} algorithm for \textit{RFM}-\textit{ANN}s, which we coin \textit{S-SWIM}. We provide a thorough theoretical discussion and supplementary numerical experiments showing that \textit{S-SWIM} can reach high accuracies on time series forecasting as a standalone strategy and serve as an effective initialisation strategy before gradient-based training. Additional ablation studies show that our proposed method performs better than random sampling of network weights.
BibTeX
@misc{gollwitzer2025random,
  author = {Maximilian Gollwitzer and Felix Dietrich},
  title = {{Random Feature Spiking Neural Networks}},
  year = {2025},
  eprint = {2510.01012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01012},
}
Figure