Improving Classification Accuracy of Feedforward Neural Networks for Spiking Neuromorphic Chips

Antonio Jimeno-Yepes, Jianbin Tang, Benjamin Scott MashfordIJCAI 2017 · DOI

TL;DR

This paper introduces a novel training algorithm that directly configures neuromorphic hardware like IBM's TrueNorth for improved classification accuracy. By training a binary crossbar and proposing a new neuron model, they achieve state-of-the-art performance on MNIST and significantly improve accuracy on EEG data compared to previous neuromorphic approaches.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) excel at tasks like image analysis but are power-hungry, typically running on GPUs. Neuromorphic chips, such as IBM's TrueNorth, offer a compelling alternative by drastically reducing power consumption (to milliwatts) while enabling real-time processing. However, a major hurdle is the incompatibility between the continuous nature of traditional DNNs and the discrete, spiking nature of neuromorphic hardware. Previous methods attempted to bridge this gap by training DNNs for continuous probabilities and then "sampling" these into discrete configurations for neuromorphic deployment, often leading to a performance drop. This paper aims to overcome this limitation by developing a training method that directly produces configurations suitable for neuromorphic deployment, thereby improving accuracy.

Methods

The core contribution is a new learning algorithm designed to directly train a neuromorphic hardware configuration. Instead of training a standard DNN and then converting it, this method trains a binary crossbar that directly accommodates the constraints of the TrueNorth architecture. They also propose a modified neuron model.

Key methodological aspects include:

  • Direct Binary Crossbar Training: The algorithm learns the binary crossbar connections (0 or 1) that are fundamental to neuromorphic hardware like TrueNorth. This contrasts with prior work that learned connection probabilities and then sampled them.
  • Effective Weights: The integration function in the neuromorphic neuron is defined by both axon weights ($s_{ij}$) and the binary crossbar connections ($c_{ij}$), where the effective weight is $w_{ij} = c_{ij} s_{ij}$. The training focuses on learning these binary crossbar connections.
  • Spiking Neuron Model: A spiking neuron model is used, where the output is binary (0 or 1, representing no spike or a spike). To enable backpropagation, a smooth approximation of the Heaviside step function (using a logistic function) is employed.
  • Constrained Training: The training process explicitly incorporates the constraints of neuromorphic hardware, such as limited weight values and connectivity.
  • Rate Coding: For deployment, data is encoded into spikes using a rate code scheme, where the number of spikes within a time window represents the data value.

Datasets

The paper evaluates their approach on two datasets:

  • MNIST: A standard benchmark dataset for handwritten digit recognition (28x28 grayscale images).
  • EEG Data: Data from a hand squeeze task, where EEG signals are processed into 46x46 grayscale images. This dataset was used to compare against previous work on neuromorphic platforms.

Key Insights and Ideas

  • Direct Training is Key: Training directly for the neuromorphic hardware constraints, rather than converting from a continuous model, significantly improves accuracy. This suggests that the "sampling" step in previous methods was a bottleneck.
  • Binary Crossbar Learning: The ability to directly learn the binary crossbar connections is crucial for efficient mapping to hardware and achieving high performance.
  • Effective Weight Configuration Matters: The choice of axon type weights ($s_{ij}$) impacts performance, with a more restricted set ($s = [-1,1]$) yielding better results for EEG data.
  • Normalization Benefits Prediction: Normalizing input features at the feature level improves the performance of trained models, indicating the importance of data preprocessing.
  • Hardware Constraints Inform Design: The paper highlights how understanding hardware limitations (e.g., number of axons per core) is essential for designing effective neural network architectures for neuromorphic chips.

Key Results

  • EEG Data: Achieved an accuracy of 86%, a significant improvement from the 76% reported by previous neuromorphic work [Nurse et al., 2016] and comparable to unconstrained neural networks [Nurse et al., 2015].
  • MNIST Data: Achieved state-of-the-art performance, comparable to non-constrained neural networks, with the large network configuration showing better results.
  • Hardware Utilization: The trained models utilize a small fraction of the TrueNorth chip (e.g., 30 cores out of 4096), indicating potential for highly power-efficient applications.
  • Weight Distribution: Analysis of effective weights and biases revealed that approximately half of the weights are zero, suggesting that many crossbar connections are disconnected, which could inform further hardware design constraints.

Limitations and Future Work

  • Limited Neuron Model: The paper uses a simplified spiking neuron model. Future work could explore more complex neuron models.
  • Hardware-Specific: While the method is presented with TrueNorth as an example, generalizing it to other neuromorphic architectures is a potential avenue.
  • Expanding to Other Architectures: The authors express interest in extending this methodology to train other deep learning algorithms, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).
  • Hardware Design Insights: Further analysis of learned parameters could provide insights for hardware design to improve crossbar usage and overall efficiency.

Summary updated

BibTeX
@inproceedings{jimenoyepes2017improving,
  author = {Antonio Jimeno-Yepes and Jianbin Tang and Benjamin Scott Mashford},
  title = {{Improving Classification Accuracy of Feedforward Neural Networks for Spiking Neuromorphic Chips}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/274},
}
Figure