TL;DR
This paper provides a mathematical framework to characterize and optimize the generalizability of Spiking Neural Networks (SNNs) trained with unsupervised Spike Timing Dependent Plasticity (STDP). By modeling the learning process as a stochastic fractal and using the Hausdorff dimension as a metric, the authors develop a Bayesian optimization approach that significantly reduces generalization error with minimal impact on classification accuracy.
What Problem Does This Paper Address?
In machine learning, "generalizability" refers to how well a model performs on new, unseen data compared to its training data. While this is well-studied for standard Deep Neural Networks (DNNs) trained with Stochastic Gradient Descent (SGD), it remains poorly understood for SNNs.
SNNs are neuro-inspired models that communicate via discrete spikes, and they often learn using STDP—a biological rule where synaptic weights change based on the relative timing of pre- and post-synaptic spikes. The challenge is that measuring generalization typically requires a separate validation set, which is computationally expensive to evaluate during the hyperparameter tuning phase. Furthermore, there has been no formal mathematical model to explain how STDP hyperparameters (like learning rates or scaling factors) actually influence an SNN's ability to generalize.
Methods
The authors propose a novel approach by treating the STDP learning process as a stochastic process (specifically, an Ornstein-Uhlenbeck process). They leverage tools from fractal geometry to analyze the trajectories of synaptic weights during training:
- Hausdorff Dimension (HD): They use HD to measure the "roughness" or complexity of the paths the weights take through the landscape. A lower HD indicates a "smoother" trajectory that tends to find flatter, more robust minima.
- Blumenthal-Getoor (BG) Index: Since HD is hard to compute directly, they use the BG index as an upper bound. This index is linked to the "tail behavior" of the weight distribution; heavier tails (more extreme weight values) often correlate with better generalization.
- Bayesian Optimization: They employ Sequential Model-based Bayesian Optimization (SMBO) to search for hyperparameters that minimize the BG index. This allows them to optimize for generalization during the training process without needing constant inference on a test set.
- STDP Variants: They compare three learning rules: additive (add-STDP), multiplicative (mult-STDP), and logarithmic (log-STDP).
Datasets
The researchers evaluated their framework on two standard benchmarks:
- MNIST: The classic dataset of 70,000 handwritten digits.
- Fashion-MNIST: A more complex dataset of 70,000 fashion product images, which serves as a more difficult test for generalization due to higher intra-class variance.
The SNN architecture used for these tasks is a two-layer network with 6,400 excitatory neurons, as illustrated in the paper's architecture diagram:
Key Insights and Ideas
- Log-STDP is Superior: The authors find that log-STDP naturally produces a log-normal weight distribution with heavier tails compared to additive or multiplicative rules. This leads to a lower Hausdorff dimension and, consequently, better generalization.
- The Generalization-Trainability Trade-off: A key "aha moment" is the discovery of a fundamental trade-off. Hyperparameters that improve generalization (making the model robust to new data) often make the model "harder" to train, leading to slower convergence and higher training loss.
- Heavy Tails = Better Generalization: The paper demonstrates that as the stochastic process becomes "heavier-tailed" (lower $\alpha$ value in the Lévy process), the Hausdorff dimension decreases, leading to a more generalizable model. This is visualized through the probability density functions of the weight updates:
Key Results
- Quantifiable Improvement: Using Bayesian Optimization on the log-STDP model for MNIST, the authors achieved a testing accuracy of 90.65% with a generalization error of only 3.17.
- Performance Gain: This represents a nearly 40% increase in generalization performance compared to randomly initialized hyperparameters, for a negligible drop (~1%) in raw testing accuracy.
- Scaling Factors: The ratio of potentiation to depression (scaling function ratio) was found to be critical. A smaller ratio leads to a lower Hausdorff dimension and better generalization. The differences in how these rules scale weights are shown below:
Limitations and Future Work
While the paper provides a robust theoretical foundation, it notes several areas for expansion:
- Deterministic vs. Stochastic STDP: The current study focuses on deterministic STDP rules. Future work could explore stochastic STDP, where weight updates are probabilistic.
- Architectural Complexity: The experiments were conducted on relatively simple two-layer fully connected SNNs. Testing this framework on Convolutional SNNs (CSNNs) or deeper architectures would be a logical next step.
- Neuron Dynamics: The authors suggest that future optimization could couple STDP hyperparameters with the internal dynamics of the neurons themselves (e.g., refractory periods and membrane thresholds) to further refine performance.
Summary updated