Abstract
A Spiking Neural Network (SNN) is trained with Spike Timing Dependent Plasticity (STDP), which is a neuro-inspired unsupervised learning method for various machine learning applications. This paper studies the generalizability properties of the STDP learning processes using the Hausdorff dimension of the trajectories of the learning algorithm. The paper analyzes the effects of STDP learning models and associated hyper-parameters on the generalizability properties of an SNN. The analysis is used to develop a Bayesian optimization approach to optimize the hyper-parameters for an STDP model for improving the generalizability properties of an SNN.
Introduction
While deep learning has well-established theories for generalization—the ability of a model to perform on unseen data—Spiking Neural Networks (SNNs), particularly those employing biologically inspired, unsupervised rules like Spike Timing Dependent Plasticity (STDP), remain poorly understood. In standard neural networks, generalization is often analyzed through the "flatness" of the loss landscape. However, STDP does not follow a global loss gradient; it is a local, noisy, and unsupervised process. This paper addresses the challenge of quantifying "overfitting" in these neuro-inspired systems by proposing a mathematical proxy for generalization that does not require a labeled test set during the hyperparameter tuning phase.
Methodology and STDP Variants
The researchers bridge the gap between stochastic calculus and neuromorphic engineering by modeling the STDP weight update process as an Ornstein-Uhlenbeck (O-U) process, a specific type of Lévy process. This accounts for the inherent randomness in biological spiking patterns.
The study compares three primary STDP learning rules:
- addSTDP: Weight-independent updates.
- multSTDP: Updates that scale linearly with synaptic weight.
- logSTDP: Updates following a logarithmic dependence, mimicking biological observations of log-normal weight distributions.
As shown in Figure 5, these rules lead to significantly different weight distributions. LogSTDP, in particular, produces heavy-tailed distributions that are mathematically linked to more stable generalization and lower geometric complexity in the learning path.
Fractal Analysis and the "Roughness" Proxy
Central to the paper's theoretical framework is the hypothesis that the trajectories of synaptic weights during learning are random fractals. The authors use the Hausdorff dimension (HD)—specifically bounded by the Blumenthal-Getoor (BG) index—to measure the "roughness" of these trajectories.
Figure 6 illustrates how varying the stability parameter $\alpha$ changes the geometric complexity of the path. A "rougher" trajectory (higher HD) suggests the model is capturing excessive noise from the training data, leading to poorer generalization. By minimizing this "roughness" proxy, the authors aim to find hyperparameters that yield smoother, more generalizable learning paths. This allows for optimization without needing a validation set during the initial search phase.
Key Results and Optimization
To automate the search for optimal parameters, the authors employed Sequential Model-based Bayesian Optimization (SMBO) to minimize the BG index. The approach was validated on the MNIST and Fashion-MNIST datasets with significant results:
- Generalization Boost: Minimizing the Hausdorff dimension reduced the generalization error on MNIST from 5.79 to 3.17—a ~45% improvement—while maintaining a high testing accuracy of 90.65%.
- Consistency Across Rules: Across all STDP variants, a lower fractal dimension consistently correlated with a smaller gap between training and testing accuracy. On the more complex Fashion-MNIST dataset, logSTDP achieved a generalization error of ~9.93 compared to ~11.73 for addSTDP.
- Optimization Efficiency: The Bayesian approach converged to optimal parameters within approximately 50 functional evaluations, proving far more efficient than brute-force grid searches.
Conclusion
This research demonstrates that the geometric complexity (fractal dimension) of the learning path itself is a powerful predictor of generalization in SNNs. Log-STDP emerged as a superior rule due to its ability to produce stable, heavy-tailed weight distributions. While the study focused on relatively simple two-layer architectures, the findings provide a robust theoretical foundation for scaling unsupervised learning in deeper, more complex neuromorphic systems.
Summary updated