TL;DR
This paper introduces MONAS-ESNN, a novel multi-objective neural architecture search (NAS) framework that efficiently discovers Spiking Neural Network (SNN) architectures optimizing both accuracy and energy efficiency. By employing a training-free NAS approach and a new metric called Adjusted Sparsity-Aware Hamming Distance (ASAHD), MONAS-ESNN significantly reduces computational cost and identifies SNNs that outperform existing methods in accuracy and spike reduction.
What Problem Does This Paper Address?
Traditional Artificial Neural Networks (ANNs) are energy-intensive. Spiking Neural Networks (SNNs) offer a biologically plausible and energy-efficient alternative, mimicking brain-like processing with sparse, event-driven spikes. However, current SNN development often relies on converting ANNs to SNNs, which fails to fully leverage SNNs' unique temporal dynamics and can lead to inefficiencies. Designing SNN-specific architectures manually is challenging and time-consuming. While Neural Architecture Search (NAS) has automated ANN design, its application to SNNs is nascent, with existing methods often being computationally expensive or not fully optimizing for both accuracy and the crucial energy efficiency (measured by spike count) of SNNs. This paper addresses the need for an efficient NAS method that specifically designs SNN architectures to maximize accuracy while minimizing energy consumption.
Methods
The core of MONAS-ESNN is a three-phase approach:
- Macro and Micro Architecture Definition: A macro-level architecture is defined to capture essential SNN components, and a search space for micro-level neuron cell candidates is established. This space includes operations like zeroing, skip connections, and spiking convolutions with varying kernel sizes, importantly incorporating forward and backward connections.
- Training-Free NAS with ASAHD: Instead of training each candidate architecture, MONAS-ESNN uses a training-free NAS approach. It evaluates architectures based on the temporal activation patterns of their initialized network. A key innovation is the Adjusted Sparsity-Aware Hamming Distance (ASAHD), a new metric designed to better capture the variability in spike activation patterns, especially for adaptable neurons like Parametric Leaky Integrate-and-Fire (PLIF) neurons. ASAHD enhances the scoring mechanism by considering the average sparsity across layers and samples.
- Multi-Objective Optimization with NSGA-II: The Non-dominated Sorting Genetic Algorithm II (NSGA-II) is employed to optimize two conflicting objectives: maximizing accuracy and minimizing the number of generated spikes. NSGA-II generates a Pareto front of optimal solutions, representing the best trade-offs between these objectives. The best micro-level cells identified are then integrated into the pre-designed macro architecture and fully trained.
Datasets
The proposed MONAS-ESNN was evaluated on three standard image classification benchmarks:
- CIFAR-10
- CIFAR-100
- Tiny-ImageNet-200
No new benchmarks were introduced; the focus was on evaluating the NAS approach on existing datasets.
Key Insights and Ideas
- Training-free NAS is viable for SNNs: The paper demonstrates that evaluating untrained network activations can effectively predict performance, drastically reducing computational cost compared to training-based NAS.
- ASAHD is crucial for PLIF neurons: The adaptability of PLIF neurons introduces significant variability in sparsity. The proposed ASAHD metric is specifically designed to handle this, leading to more accurate architecture evaluation than previous methods like SAHD.
- Multi-objective optimization is key for SNNs: Balancing accuracy and spike count (energy efficiency) through NSGA-II yields better overall SNN designs than optimizing for accuracy alone.
- Backward connections matter: The architecture search space explicitly includes backward connections, which are shown to be beneficial for SNN performance.
Key Results
- MONAS-ESNN identified SNN architectures that achieved higher accuracy with fewer generated spikes and fewer timesteps compared to existing state-of-the-art methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet-200.
- For CIFAR-10 with 16 channels, MONAS-ESNN achieved 89.81% accuracy with only 94k spikes in 3 timesteps, outperforming other methods.
- The ASAHD metric improved the correlation between architecture scores and actual accuracy, as evidenced by higher Kendall's τ values (e.g., 0.6153 for LIF neurons, 0.6538 for PLIF neurons, compared to SAHD's 0.5769 and 0.3230 respectively).
- The proposed model achieved competitive accuracy with significantly reduced spike counts, demonstrating improved energy efficiency.
Limitations and Future Work
- Computational Cost: While significantly reduced by training-free NAS, the optimization process still involves considerable computation, especially with a large population size and many generations for NSGA-II.
- Generalizability: The current evaluation is limited to image classification tasks.
- Future Work: The authors suggest expanding the evaluation to more diverse datasets and real-world applications. Further exploration of integrating PLIF neurons with backward connections and optimizing other micro-level components could also be beneficial.
Summary updated