Abstract
Deep Neural Networks (DNNs) are often pruned to reduce computational costs for edge deployment, resulting in sparse weight matrices. However, traditional hardware architectures struggle to efficiently skip zero-valued computations without significant metadata overhead. This paper presents the Nested Bitmask Neural Network (NBNN), a specialized FPGA-based framework that utilizes a two-tier bitmask structure to encode and bypass zeros at both block and element levels. Experimental results on a sparse LeNet-5 model demonstrate a 15x speedup compared to standard FPGA implementations and superior energy efficiency relative to high-end GPUs.
Introduction
The deployment of Deep Neural Networks (DNNs) on resource-constrained edge devices necessitates optimization techniques like pruning. Pruning removes redundant connections, transforming dense matrices into sparse ones. While sparsity offers theoretical speedups, practical gains are often hindered by hardware limitations. Conventional CPUs and GPUs typically process zero-valued weights as standard operands, leading to wasted cycles and energy. FPGAs provide the flexibility to design custom logic for skipping zeros, but the challenge lies in managing the "metadata"—the indices tracking non-zero values—without incurring excessive storage or processing costs.
Methodology
The proposed Nested Bitmask Structure addresses metadata overhead through a hierarchical approach. The weight matrix is first partitioned into uniform blocks, and then processed via a two-level system:
- Level 1 (Block-level) Bitmask: A coarse-grained binary map indicates whether a block contains any non-zero elements. If a block is entirely zero, the hardware skips it immediately.
- Level 2 (Element-level) Bitmask: For blocks with non-zero values, a secondary bitmask identifies the specific locations of these values. Only non-zero elements are streamed to the Processing Engines (PEs).
The hardware implementation on a Xilinx Virtex-7 FPGA leverages FIFOs for data streaming and custom branch prediction logic to manage the bitmask-driven execution flow. This nested design minimizes the address data required, avoiding the "large stride" issues common in relative indexing schemes.
Key Insights: Pruning Strategies and Efficiency
The effectiveness of the NBNN framework is closely tied to the nature of the sparsity. The authors distinguish between "Edge Pruning," which simply nullifies individual weights, and "Neuron Pruning," which alters the network's structural connectivity to be more hardware-friendly.
A critical finding is the "sweet spot" for nested bitmasks: the framework significantly outperforms single-layer bitmasks when sparsity exceeds 90%. At this high level of sparsity, the probability of skipping entire blocks increases, maximizing the benefits of the hierarchical structure. By using bitmasks instead of relative indices, the design avoids storing long gaps between non-zero values, which typically requires more bits.
Results and Evaluation
The framework was evaluated using a sparse LeNet-5 model, typically benchmarked on the MNIST dataset. The focus was on Fully Connected (FC) layers, where sparsity is most prevalent.
- Latency Reduction: The NBNN implementation achieved a latency of 10.76ms, a 15x improvement over the 161.01ms recorded for a non-optimized FPGA baseline.
- Energy Efficiency: Power consumption was measured at 21.65W, far lower than the 120W typical of a mid-range GPU (e.g., GTX 1060), highlighting its suitability for power-sensitive environments.
- Resource Utilization: The nested structure proved more resource-efficient than vanilla multi-PE approaches, requiring fewer BRAM and DSP units.
Conclusion
The Nested Bitmask Neural Network framework provides a robust solution for accelerating sparse DNN inference on FPGAs. By optimizing metadata representation and hardware execution flow, it achieves significant gains in both speed and energy efficiency. Future research will focus on extending this approach to Convolutional (CONV) layers and scaling the evaluation to larger, more modern architectures like ResNet.
Summary updated