TL;DR
This paper introduces a Bayesian optimization approach to fine-tune hyperparameters for Whetstone, a training algorithm for binary communication networks suitable for neuromorphic hardware. By automatically optimizing these parameters, the authors achieve significant accuracy improvements over previous state-of-the-art results on several benchmark datasets, demonstrating the potential of systematic hyperparameter tuning for neuromorphic systems.
What Problem Does This Paper Address?
Training neural networks for deployment on neuromorphic hardware presents unique challenges. These systems often utilize binary communication and spiking neuron models, which behave differently from traditional artificial neural networks. Consequently, standard training algorithms need adaptation, and determining the optimal hyperparameters (settings that control the learning process) for these adapted algorithms becomes a complex, non-trivial task. Traditional methods like trial-and-error or random search are inefficient and may not yield the best performance. This paper addresses the gap in systematically finding optimal hyperparameters for such neuromorphic-friendly training algorithms, specifically focusing on the Whetstone algorithm.
Methods
The core of this paper is the application of Bayesian Optimization to tune the hyperparameters of the Whetstone training algorithm.
- Whetstone: This algorithm trains networks with binary communication, making them suitable for spiking neuromorphic hardware. It starts with differentiable activation functions (like sigmoids) and gradually "sharpens" them into non-differentiable threshold functions during training. This process introduces several hyperparameters related to the sharpening schedule (when it starts, how long it lasts, and the intervals between sharpening steps), in addition to standard network training hyperparameters.
- Bayesian Optimization: This is a sequential, model-based optimization technique designed for expensive, black-box objective functions (like training a neural network and evaluating its accuracy). It builds a probabilistic model (a Gaussian process) of the objective function based on observed data points. This model is then used to select the next set of hyperparameters to evaluate, balancing exploration (trying new, uncertain regions of the search space) and exploitation (focusing on regions likely to yield good results). The authors use a single-objective Bayesian optimization approach to maximize classification accuracy.
Datasets
The authors evaluated their approach on four commonly used image classification datasets:
- MNIST: A dataset of handwritten digits (0-9), 28x28 pixels.
- Fashion MNIST: A dataset of clothing items, designed as a drop-in replacement for MNIST, also 28x28 pixels.
- CIFAR-10: A dataset of color images of common objects (dogs, airplanes, etc.), 32x32 pixels, with 10 classes.
- CIFAR-100: Similar to CIFAR-10 but with 100 classes.
Key Insights and Ideas
- Hyperparameter Sensitivity: The performance of neuromorphic-friendly training algorithms like Whetstone is highly sensitive to hyperparameter choices, often more so than traditional neural networks.
- Dataset-Specific Optimization: The optimal set of hyperparameters is not universal; it varies significantly across different datasets, underscoring the need for per-dataset optimization.
- Bayesian Optimization Efficiency: Bayesian optimization can find near-optimal hyperparameters with significantly fewer evaluations compared to grid search, making it practical for expensive training processes.
- Potential for Improvement: Even with existing algorithms like Whetstone, systematic hyperparameter optimization can unlock substantial performance gains, suggesting that many off-the-shelf neuromorphic algorithms could be improved.
Key Results
- Accuracy Improvements: The Bayesian optimization approach achieved accuracy improvements of up to 15% over the previous state-of-the-art for Whetstone on the tested datasets.
- For CIFAR-100, accuracy increased from 38% to 53.13% (Case Study One).
- For MNIST, accuracy improved from 99.53% to 99.6% (Case Study Two).
- For Fashion-MNIST, accuracy increased from 93.2% to 93.68% (Case Study Two).
- For CIFAR-10, accuracy increased from 79% to 84.36% (Case Study Two).
- Efficiency: In Case Study One, Bayesian optimization found an almost optimal hyperparameter combination after only 15 evaluations, whereas grid search required 256 evaluations to achieve a comparable result.
- Sensitivity Analysis: The paper demonstrates how changes in specific hyperparameters (e.g., learning rate, sharpening schedule) can drastically impact performance, and how Bayesian optimization navigates these sensitivities.
Limitations and Future Work
- Network Architecture: While some CNN architecture hyperparameters (filter sizes, number of features) were optimized, the paper acknowledges that further optimization of network architecture (e.g., depth, layer types) using methods like MENNDL could yield even better results.
- Hardware Mapping: The current work focuses on optimizing the training algorithm itself, not the subsequent mapping of the trained network to specific neuromorphic hardware. Future work plans to incorporate hardware performance metrics (e.g., power, area) into the optimization objective and to evaluate performance on actual neuromorphic platforms like SpiNNaker or Loihi.
- Multi-Objective Optimization: The authors suggest that problems with similar classification performance but different resource requirements (energy, area) might benefit from multi-objective optimization, which they have explored in prior work and plan to apply to Whetstone.
Summary updated