DARB: A Density-Adaptive Regular-Block Pruning for Deep Neural Networks

Ao Ren, Tao Zhang, Yuhao Wang, Sheng Lin, Peiyan Dong, Yen-Kuang Chen, Yuan Xie, Yanzhi Wang

Abstract

The rapidly growing parameter volume of deep neural networks (DNNs) hinders artificial intelligence applications on resource-constrained devices. Neural network pruning is a mainstream model compression technique used to reduce model size and computation. Unlike irregular pruning, which incurs high index storage and decoding overhead, or structured pruning, which often fails to analyze sparse network characteristics deeply, we propose Density-Adaptive Regular-Block (DARB) pruning. By examining structural characteristics like row redundancy and sensitivity, DARB employs a novel block-max weight masking (BMWM) method and density-adaptive block sizing. Our results show DARB achieves 13× to 25× pruning ratios—a 2.8× to 4.3× improvement over state-of-the-art counterparts—with 14.3× higher decoding efficiency.

Introduction

Deep Neural Networks (DNNs) are notoriously memory-intensive and computationally expensive, making them difficult to deploy on edge devices like smartphones. Pruning—removing unimportant weights—is a standard solution, but researchers face a "Goldilocks" problem: irregular pruning achieves high compression but creates a "messy" sparse matrix that is slow to decode, while structured pruning is hardware-friendly but often forces the removal of important weights, leading to accuracy loss.

The authors argue that prior structured pruning methods ignore the intrinsic characteristics of the network, such as the fact that different rows in a weight matrix have vastly different levels of redundancy. DARB bridges this gap by adapting block sizes to specific row characteristics.

Methodology

DARB utilizes a two-step approach based on the study of irregularly pruned matrices:

  1. Block-Max Weight Masking (BMWM): Instead of picking the top weights globally, rows are divided into equal blocks, and only the weight with the largest magnitude in each block is retained. These "locally salient" weights are nearly as important as globally salient ones but much easier for hardware to index.
  2. Density-Adaptive Regular-Block (DARB) Pruning: Recognizing that some rows are naturally "denser" (more important) than others, DARB allows different rows to have different block sizes. To keep it hardware-friendly, block sizes must be a power of two (e.g., 2, 4, 8, 16).
Hardware-efficient decoding process using DARB's regular-block structure.

As shown in the hardware architecture above, DARB's structure allows for parallel decoding. Because it uses small, fixed-size blocks within a row, hardware can use multiple small multiplexers to fetch weights in parallel, avoiding the sequential bottleneck of traditional Compressed Sparse Row (CSR) formats.

Key Insights and Ideas

A critical observation driving this work is Row Density Variance. In a naturally pruned network, the densest rows can have up to 14× more weights than the sparsest rows.

Density distribution of weights across different layers, showing significant variance in row redundancy.

As illustrated in the histograms above, forcing every row to have the same sparsity is fundamentally inefficient. The authors discovered that dense rows are much more sensitive to further pruning than sparse rows. Maintaining this variety of row density is the secret to sustaining model accuracy at high compression levels.

Key Results

  • High Pruning Ratios: DARB achieved 13× to 25× pruning ratios on LSTM models, representing a 2.8× to 4.3× improvement over state-of-the-art structured pruning methods like ISS and BBS.
  • Accuracy Preservation: On ImageNet with AlexNet, DARB achieved a 21.3× pruning ratio with zero accuracy loss compared to the baseline.
  • Hardware Efficiency: DARB demonstrated 14.3× higher decoding efficiency than standard block pruning due to its power-of-two structure allowing for massive hardware parallelism.
  • Versatility: The method also showed strong results on Convolutional Neural Networks (CNNs), achieving 18.7× pruning on VGG-16 with only 0.6% accuracy degradation.

Limitations and Future Work

While DARB is highly effective for fully-connected layers, future work aims to further optimize it for the unique structural requirements of convolutional layers in state-of-the-art CNN architectures. Additionally, the current workflow requires an initial "trial" irregular pruning to determine row densities, which the authors hope to streamline in future iterations.

Summary updated

Abstract

The rapidly growing parameter volume of deep neural networks (DNNs) hinders the artificial intelligence applications on resource constrained devices, such as mobile and wearable devices. Neural network pruning, as one of the mainstream model compression techniques, is under extensive study to reduce the model size and thus the amount of computation. And thereby, the state-of-the-art DNNs are able to be deployed on those devices with high runtime energy efficiency. In contrast to irregular pruning that incurs high index storage and decoding overhead, structured pruning techniques have been proposed as the promising solutions. However, prior studies on structured pruning tackle the problem mainly from the perspective of facilitating hardware implementation, without diving into the deep to analyze the characteristics of sparse neural networks. The neglect on the study of sparse neural networks causes inefficient trade-off between regularity and pruning ratio. Consequently, the potential of structurally pruning neural networks is not sufficiently mined.In this work, we examine the structural characteristics of the irregularly pruned weight matrices, such as the diverse redundancy of different rows, the sensitivity of different rows to pruning, and the position characteristics of retained weights. By leveraging the gained insights as a guidance, we first propose the novel block-max weight masking (BMWM) method, which can effectively retain the salient weights while imposing high regularity to the weight matrix. As a further optimization, we propose a density-adaptive regular-block (DARB) pruning that can effectively take advantage of the intrinsic characteristics of neural networks, and thereby outperform prior structured pruning work with high pruning ratio and decoding efficiency. Our experimental results show that DARB can achieve 13× to 25× pruning ratio, which are 2.8× to 4.3× improvements than the state-of-the-art counterparts on multiple neural network models and tasks. Moreover, DARB can achieve 14.3× decoding efficiency than block pruning with higher pruning ratio.
BibTeX
@inproceedings{ren2020darb,
  author = {Ao Ren and Tao Zhang and Yuhao Wang and Sheng Lin and Peiyan Dong and Yen-Kuang Chen and Yuan Xie and Yanzhi Wang},
  title = {{DARB: A Density-Adaptive Regular-Block Pruning for Deep Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=bazihHx0SS},
}
Figure