TL;DR
This paper introduces a novel VLSI architecture for accelerating Spiking Convolutional Neural Networks (SCNNs) for real-time inference in embedded systems. By cleverly representing SCNN operations as time-step binary spike maps, they achieve significant reductions in hardware complexity and power consumption while maintaining high processing speeds and accuracy.
What Problem Does This Paper Address?
Neuromorphic computing, inspired by the human brain, promises highly energy-efficient AI. Spiking Neural Networks (SNNs) are a key component of this, communicating information through sparse electrical spikes, mimicking biological neurons. However, traditional hardware like CPUs and GPUs are ill-suited for the temporal and sparse nature of SNNs. While dedicated neuromorphic hardware exists, existing designs often fail to fully exploit the inherent structure of Convolutional Neural Networks (CNNs) when adapted to spiking neurons (SCNNs). Specifically, they may waste resources by replicating shared convolutional weights or using inefficient communication protocols like Address Event Representation (AER). This paper tackles the challenge of designing a cost-efficient, high-speed VLSI architecture for SCNN inference that overcomes these limitations, making SNNs more practical for real-world embedded applications.
Methods
The core innovation lies in the proposed "time-step binary spike map" concept. Instead of complex AER packets, the output spikes of neurons in a CONV layer at each discrete time-step are captured as a binary image (spike map). This allows the SCNN inference to be decomposed into a series of regular, time-step CNN-like operations. The architecture then employs a "pixel stream processing mechanism" where these binary spike maps are processed sequentially, pixel by pixel, but in a channel-parallel manner. This approach significantly reduces hardware resource requirements by:
- Minimizing Memory Footprint: Spike maps are processed "on the fly," requiring only small buffers for recent rows and columns, crucial for on-chip implementation.
- Maximizing Pixel Throughput: A pipelined architecture allows for high processing speeds, aiming for up to one pixel per channel per clock cycle.
- Reducing Computing Resources: Simple bitwise operations and adders replace expensive multipliers for weight-product calculations, and a single processing circuit can be reused across channels.
The architecture includes an input spike map generator, groups of CONV modules, FC modules, pixel stream row buffers, and an output spike counter. The CONV and FC modules are designed with pipelined adder trees, bias addition, and a Heaviside function evaluation, all optimized for binary spike map processing.
Datasets
The paper uses two standard image datasets for evaluation:
- MNIST: A dataset of 70,000 grayscale images of handwritten digits (0-9).
- Fashion-MNIST: A more challenging dataset of 70,000 grayscale images of clothing items, also belonging to 10 object categories.
The SCNN models were trained offline using an equivalent non-spiking CNN model and then converted to SCNN models using a method described in prior work.
Key Insights and Ideas
- Binary Spike Maps as a Game Changer: Representing SCNN states as binary maps at each time-step is a crucial simplification. It allows leveraging well-established CNN processing techniques, drastically reducing hardware complexity compared to AER-based methods.
- Pixel Stream Processing for Efficiency: Processing data as a stream, rather than entire frames or maps, is key to minimizing memory usage and enabling high throughput in a cost-effective manner.
- Scalability is Key: The architecture's modular design and the demonstration with different CONV/FC layer configurations highlight its scalability for deeper SCNN models, with the 3C1F configuration showing improved accuracy.
- FPGA as a Viable Prototyping Platform: The successful implementation and evaluation on a Xilinx Zynq-7045 FPGA demonstrate the practical feasibility of the proposed architecture for real-world embedded systems.
Key Results
The proposed architecture was prototyped on a Xilinx Zynq-7045 FPGA and achieved:
- High Processing Speed: Up to 1250 frames/s on both MNIST and Fashion-MNIST datasets at a 100 MHz clock frequency.
- Competitive Recognition Accuracies: Achieved 97.3% accuracy on MNIST and 83.3% accuracy on Fashion-MNIST with a 3 CONV layer and 1 FC layer configuration (3C1F).
- Moderate Power Consumption: The 3C1F configuration consumed approximately 1.241W.
- Significant Speedup: Achieved processing accelerations of 1103x, 998x, and 638x compared to software-based SNN/SCNN implementations on MNIST.
Limitations and Future Work
- Offline Training: The current SCNN models are trained offline, and the learned weights are downloaded to the hardware. The paper mentions that future work could explore incorporating online learning capabilities directly on the hardware.
- Comparison to Advanced Training: While the achieved accuracies are competitive, the paper notes that a more advanced CNN-to-SCNN training methodology (from reference [41]) could potentially yield even higher accuracies, which they plan to investigate.
- Specific SCNN Models: The architecture is optimized for feedforward SCNN models, which are widely used. Future work might explore adapting it for more complex SNN architectures.
- Custom ASIC vs. FPGA: The paper acknowledges that custom ASIC implementations could offer even higher energy efficiency and lower power consumption, but at the cost of longer design cycles and less flexibility for upgrades compared to the FPGA prototype.
Summary updated