Accelerated Inference Framework of Sparse Neural Network Based on Nested Bitmask Structure

Yipeng Zhang, Bo Du, Lefei Zhang, Rongchun Li, Yong Dou

Abstract

Deep Neural Networks (DNNs) are often pruned to reduce computational costs for edge deployment, resulting in sparse weight matrices. However, traditional hardware architectures struggle to efficiently skip zero-valued computations without significant metadata overhead. This paper presents the Nested Bitmask Neural Network (NBNN), a specialized FPGA-based framework that utilizes a two-tier bitmask structure to encode and bypass zeros at both block and element levels. Experimental results on a sparse LeNet-5 model demonstrate a 15x speedup compared to standard FPGA implementations and superior energy efficiency relative to high-end GPUs.

Introduction

The deployment of Deep Neural Networks (DNNs) on resource-constrained edge devices necessitates optimization techniques like pruning. Pruning removes redundant connections, transforming dense matrices into sparse ones. While sparsity offers theoretical speedups, practical gains are often hindered by hardware limitations. Conventional CPUs and GPUs typically process zero-valued weights as standard operands, leading to wasted cycles and energy. FPGAs provide the flexibility to design custom logic for skipping zeros, but the challenge lies in managing the "metadata"—the indices tracking non-zero values—without incurring excessive storage or processing costs.

Methodology

The proposed Nested Bitmask Structure addresses metadata overhead through a hierarchical approach. The weight matrix is first partitioned into uniform blocks, and then processed via a two-level system:

  1. Level 1 (Block-level) Bitmask: A coarse-grained binary map indicates whether a block contains any non-zero elements. If a block is entirely zero, the hardware skips it immediately.
  2. Level 2 (Element-level) Bitmask: For blocks with non-zero values, a secondary bitmask identifies the specific locations of these values. Only non-zero elements are streamed to the Processing Engines (PEs).
The hardware architecture utilizes bitmask scanning and FIFOs to efficiently stream non-zero elements to the ALU.

The hardware implementation on a Xilinx Virtex-7 FPGA leverages FIFOs for data streaming and custom branch prediction logic to manage the bitmask-driven execution flow. This nested design minimizes the address data required, avoiding the "large stride" issues common in relative indexing schemes.

Key Insights: Pruning Strategies and Efficiency

The effectiveness of the NBNN framework is closely tied to the nature of the sparsity. The authors distinguish between "Edge Pruning," which simply nullifies individual weights, and "Neuron Pruning," which alters the network's structural connectivity to be more hardware-friendly.

Comparison of original network structure with edge pruning and neuron pruning techniques.

A critical finding is the "sweet spot" for nested bitmasks: the framework significantly outperforms single-layer bitmasks when sparsity exceeds 90%. At this high level of sparsity, the probability of skipping entire blocks increases, maximizing the benefits of the hierarchical structure. By using bitmasks instead of relative indices, the design avoids storing long gaps between non-zero values, which typically requires more bits.

Results and Evaluation

The framework was evaluated using a sparse LeNet-5 model, typically benchmarked on the MNIST dataset. The focus was on Fully Connected (FC) layers, where sparsity is most prevalent.

  • Latency Reduction: The NBNN implementation achieved a latency of 10.76ms, a 15x improvement over the 161.01ms recorded for a non-optimized FPGA baseline.
  • Energy Efficiency: Power consumption was measured at 21.65W, far lower than the 120W typical of a mid-range GPU (e.g., GTX 1060), highlighting its suitability for power-sensitive environments.
  • Resource Utilization: The nested structure proved more resource-efficient than vanilla multi-PE approaches, requiring fewer BRAM and DSP units.

Conclusion

The Nested Bitmask Neural Network framework provides a robust solution for accelerating sparse DNN inference on FPGAs. By optimizing metadata representation and hardware execution flow, it achieves significant gains in both speed and energy efficiency. Future research will focus on extending this approach to Convolutional (CONV) layers and scaling the evaluation to larger, more modern architectures like ResNet.

Summary updated

Abstract

In order to satisfy the ever-growing demand for high-performance processors for neural networks, the state-of-the-art processing units tend to use application-oriented circuits to replace Processing Engine (PE) on the GPU under circumstances where low-power solutions are required. The application-oriented PE is fully optimized in terms of the circuit architecture and eliminates incorrect data dependency and instructional redundancy. In this paper, we propose a novel encoding approach on a sparse neural network after pruning. We partition the weight matrix into numerous blocks and use a low-rank binary map to represent the validation of these blocks. Furthermore, the elements in each nonzero block are also encoded into two submatrices: one is the binary stream discriminating the zero/nonzero position, while the other is the pure nonzero elements stored in the FIFO. In the experimental part, we implement a well pre-trained sparse neural network on the Xilinx FPGA VC707. Experimental results show that our algorithm outperforms the other benchmarks. Our approach has successfully optimized the throughput and the energy efficiency to deal with a single frame. Accordingly, we contend that Nested Bitmask Neural Network (NBNN), is an efficient neural network structure with only minor accuracy loss on the SoC system.
BibTeX
@inproceedings{zhang2019accelerated,
  author = {Yipeng Zhang and Bo Du and Lefei Zhang and Rongchun Li and Yong Dou},
  title = {{Accelerated Inference Framework of Sparse Neural Network Based on Nested Bitmask Structure}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/605},
}
Figure