Abstract
The rapidly growing parameter volume of deep neural networks (DNNs) hinders artificial intelligence applications on resource-constrained devices. Neural network pruning is a mainstream model compression technique used to reduce model size and computation. Unlike irregular pruning, which incurs high index storage and decoding overhead, or structured pruning, which often fails to analyze sparse network characteristics deeply, we propose Density-Adaptive Regular-Block (DARB) pruning. By examining structural characteristics like row redundancy and sensitivity, DARB employs a novel block-max weight masking (BMWM) method and density-adaptive block sizing. Our results show DARB achieves 13× to 25× pruning ratios—a 2.8× to 4.3× improvement over state-of-the-art counterparts—with 14.3× higher decoding efficiency.
Introduction
Deep Neural Networks (DNNs) are notoriously memory-intensive and computationally expensive, making them difficult to deploy on edge devices like smartphones. Pruning—removing unimportant weights—is a standard solution, but researchers face a "Goldilocks" problem: irregular pruning achieves high compression but creates a "messy" sparse matrix that is slow to decode, while structured pruning is hardware-friendly but often forces the removal of important weights, leading to accuracy loss.
The authors argue that prior structured pruning methods ignore the intrinsic characteristics of the network, such as the fact that different rows in a weight matrix have vastly different levels of redundancy. DARB bridges this gap by adapting block sizes to specific row characteristics.
Methodology
DARB utilizes a two-step approach based on the study of irregularly pruned matrices:
- Block-Max Weight Masking (BMWM): Instead of picking the top weights globally, rows are divided into equal blocks, and only the weight with the largest magnitude in each block is retained. These "locally salient" weights are nearly as important as globally salient ones but much easier for hardware to index.
- Density-Adaptive Regular-Block (DARB) Pruning: Recognizing that some rows are naturally "denser" (more important) than others, DARB allows different rows to have different block sizes. To keep it hardware-friendly, block sizes must be a power of two (e.g., 2, 4, 8, 16).
As shown in the hardware architecture above, DARB's structure allows for parallel decoding. Because it uses small, fixed-size blocks within a row, hardware can use multiple small multiplexers to fetch weights in parallel, avoiding the sequential bottleneck of traditional Compressed Sparse Row (CSR) formats.
Key Insights and Ideas
A critical observation driving this work is Row Density Variance. In a naturally pruned network, the densest rows can have up to 14× more weights than the sparsest rows.
As illustrated in the histograms above, forcing every row to have the same sparsity is fundamentally inefficient. The authors discovered that dense rows are much more sensitive to further pruning than sparse rows. Maintaining this variety of row density is the secret to sustaining model accuracy at high compression levels.
Key Results
- High Pruning Ratios: DARB achieved 13× to 25× pruning ratios on LSTM models, representing a 2.8× to 4.3× improvement over state-of-the-art structured pruning methods like ISS and BBS.
- Accuracy Preservation: On ImageNet with AlexNet, DARB achieved a 21.3× pruning ratio with zero accuracy loss compared to the baseline.
- Hardware Efficiency: DARB demonstrated 14.3× higher decoding efficiency than standard block pruning due to its power-of-two structure allowing for massive hardware parallelism.
- Versatility: The method also showed strong results on Convolutional Neural Networks (CNNs), achieving 18.7× pruning on VGG-16 with only 0.6% accuracy degradation.
Limitations and Future Work
While DARB is highly effective for fully-connected layers, future work aims to further optimize it for the unique structural requirements of convolutional layers in state-of-the-art CNN architectures. Additionally, the current workflow requires an initial "trial" irregular pruning to determine row densities, which the authors hope to streamline in future iterations.
Summary updated