Abstract
Spiking neural networks (SNNs) are regarded as a promising candidate to deal with the major challenges of current machine learning techniques, including the high energy consumption induced by deep neural networks. However, there is still a great gap between SNNs and the few-shot learning performance of artificial neural networks. Importantly, existing spike-based few-shot learning models do not target robust learning based on spatiotemporal dynamics and superior machine learning theory. In this paper, we propose a novel spike-based framework with the entropy theory, namely, heterogeneous ensemble-based spike-driven few-shot online learning (HESFOL). The proposed HESFOL model uses the entropy theory to establish the gradient-based few-shot learning scheme in a recurrent SNN architecture. We examine the performance of the HESFOL model based on the few-shot classification tasks using spiking patterns and the Omniglot data set, as well as the few-shot motor control task using an end-effector. Experimental results show that the proposed HESFOL scheme can effectively improve the accuracy and robustness of spike-driven few-shot learning performance. More importantly, the proposed HESFOL model emphasizes the application of modern entropy-based machine learning methods in state-of-the-art spike-driven learning algorithms. Therefore, our study provides new perspectives for further integration of advanced entropy theory in machine learning to improve the learning performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic systems.
Introduction
While deep learning has mastered many tasks, it remains data-hungry and power-intensive. Spiking Neural Networks (SNNs) offer a biologically inspired, low-power alternative, but they face two major hurdles: the "Few-Shot Gap," where SNNs struggle to learn from limited examples, and a lack of robustness to non-Gaussian noise. Most learning algorithms rely on Mean Square Error (MSE), which assumes noise follows a clean Gaussian distribution. In real-world and biological environments, noise is often "heavy-tailed" or non-Gaussian, causing standard SNNs to fail. The HESFOL framework aims to bridge this gap by creating a mathematically robust, energy-efficient system for real-world uncertainty.
The HESFOL Framework
The proposed HESFOL (Heterogeneous Ensemble-based Spike-driven Few-Shot Online Learning) model introduces several core innovations to achieve robust online learning:
- Brain-Inspired Architecture: The model mimics the interaction between the hippocampus (rapid task-specific learning) and the prefrontal cortex (providing learning signals). It utilizes a two-compartment spiking neuron model (soma and dendrite) to distinguish between feedforward input and recurrent feedback, effectively addressing the "credit assignment" problem.
- Heterogeneous Ensemble Loss: HESFOL combines four loss functions to balance speed and stability: Cross-Entropy for fast initial learning, Firing Rate and Voltage Regularization for biological plausibility and energy efficiency, and the Maximum Mixture Correntropy Criterion (MMCC).
- Online Learning via e-prop: Instead of computationally expensive Backpropagation Through Time (BPTT), the model uses "eligibility propagation" (e-prop) to update weights locally and in real-time, significantly reducing memory overhead.
Robustness via Entropy Theory
The "secret sauce" of HESFOL's robustness is the integration of Correntropy. Standard MSE treats all errors equally, meaning a single massive outlier can significantly skew the model's weights. In contrast, Correntropy acts as a local similarity measure.
As shown in the comparison above, the influence function for MMSE (Minimum Mean Square Error) grows linearly with the error $e$. This makes it highly sensitive to outliers. The MCC (Maximum Correntropy Criterion) influence function, however, is bell-shaped; it increases for small errors but drops to zero for large errors. This allows the model to effectively ignore non-Gaussian "heavy-tailed" noise and outliers that would otherwise ruin the learning process.
Experimental Results
The model was evaluated on three distinct benchmarks, demonstrating superior performance and efficiency:
- Omniglot Few-Shot Classification: HESFOL achieved 93.1% accuracy on the Omniglot dataset even when subjected to non-Gaussian salt-and-pepper noise. This outperforms several existing SNN models and remains competitive with specialized ANN architectures.
- Noise Robustness: In tests with high levels of random spike deletion (up to 75% loss), HESFOL maintained significantly higher accuracy than models relying solely on Cross-Entropy or MSE.
- Energy Efficiency: The authors estimate a power consumption of only 2.6 pJ per classification task on specialized neuromorphic hardware—approximately 50,000 times more efficient than a standard GPU.
- Motor Control: The model successfully learned to control a two-joint robotic arm to trace trajectories after just 500ms of "one-shot" experience.
Conclusion
HESFOL demonstrates that integrating advanced entropy-based machine learning methods with spike-driven algorithms can significantly improve SNN performance. While future work is needed to scale this approach to high-dimensional datasets like ImageNet, HESFOL provides a robust foundation for the next generation of energy-efficient, neuromorphic learning systems.
Summary updated