Structured Deep Neural Network Pruning by Varying Regularization Parameters

Huan Wang, Qiming Zhang, Yuehai Wang, Roland Hu

Abstract

Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degrade. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fragility of the expressiveness of CNNs, and thus calls for a more gentle regularization scheme so that the networks can adapt during pruning. To achieve this, we propose a new and novel regularization-based pruning method, named IncReg, to incrementally assign different regularization factors to different weights based on their relative importance. Empirical analysis on CIFAR-10 dataset verifies the merits of IncReg. Further extensive experiments with popular CNNs on CIFAR-10 and ImageNet datasets show that IncReg achieves comparable to even better results compared with state-of-the-arts. Our source codes and trained models are available here: https://github.com/mingsun-tse/caffe_increg.

Introduction

Deep Convolutional Neural Networks (CNNs) are notoriously resource-intensive, complicating deployment on mobile and embedded devices. Pruning—the removal of redundant parameters—is a standard solution. However, existing methods often struggle with two issues: unstructured pruning creates sparse matrices that are difficult to accelerate on standard hardware, and aggressive structured pruning uses large, constant regularization factors that ignore the relative importance of different weights.

This paper introduces IncReg (Incremental Regularization), a novel structured pruning method that replaces the "heavy hammer" of constant regularization with a "gentle nudge." By incrementally adjusting regularization factors based on a weight group's relative importance, IncReg allows neural networks to adapt during the pruning process, achieving state-of-the-art compression with minimal accuracy loss.

Methodology: Incremental Regularization

The core of IncReg is the dynamic adjustment of regularization penalties. Instead of applying a fixed penalty to all weights, the algorithm follows a three-step process:

  1. Group Ranking: Weights are grouped (e.g., by filter or channel) and ranked by their $L_1$-norm, which serves as a proxy for importance.
  2. Incremental Updates: The regularization factor for each group ($\lambda_g$) is updated at each iteration.
  3. Piecewise Linear Function: A specific function determines the increment ($\Delta\lambda_g$). Groups ranked below a target pruning ratio $R$ have their penalty increased to drive them toward zero. Conversely, important groups have their penalty decreased or zeroed, allowing the network to redistribute its expressiveness to these surviving components.
The piecewise linear function used to determine the regularization increment based on group rank.

Theoretical analysis shows that slightly increasing the regularization factor at a local minimum effectively compresses weight magnitude while allowing the network to transition to a nearby, optimized local minimum.

Experimental Results

The authors evaluated IncReg on CIFAR-10 and ImageNet (ILSVRC-2012) using architectures like ResNet-50, VGG-16, and AlexNet.

Training Dynamics

A key advantage of IncReg is its "gentleness." By increasing penalties gradually, the network can transfer knowledge from dying filters to surviving ones. Figure 3 illustrates how IncReg effectively separates weight groups into those to be pruned and those to be preserved, compared to the more static behaviors of SSL and AFP.

Comparison of L1-norm evolution during training for SSL, AFP, and IncReg, showing IncReg's superior group separation.

Performance Benchmarks

IncReg consistently outperformed existing methods across various benchmarks:

  • AlexNet (ImageNet): Achieved a 4x speedup while improving top-5 accuracy by 0.2%.
  • ResNet-50 (ImageNet): Achieved a 2x speedup with a negligible 0.1% increase in error.
  • VGG-16 (ImageNet): Achieved a 5x speedup with only a 1.5% increase in error, significantly outperforming competitors that saw error spikes up to 14.6%.

Discussion and Conclusion

IncReg demonstrates that pruning should be an adaptive process. Its robustness to hyperparameter tuning—varying the main parameter $A$ by an order of magnitude resulted in only minor accuracy fluctuations—makes it a practical tool for model compression. Furthermore, by focusing on structured sparsity (rows, columns, or channels), the resulting models are directly compatible with standard hardware acceleration.

Future work includes formulating the regularization increment as a more disciplined optimization problem and extending the method to other architectures like Transformers and RNNs.

Summary updated

Abstract

Parameter pruning is a promising approach for CNN compression and acceleration by eliminating redundant model parameters with tolerable performance degrade. Despite its effectiveness, existing regularization-based parameter pruning methods usually drive weights towards zero with large and constant regularization factors, which neglects the fragility of the expressiveness of CNNs, and thus calls for a more gentle regularization scheme so that the networks can adapt during pruning. To achieve this, we propose a new and novel regularization-based pruning method, named IncReg, to incrementally assign different regularization factors to different weights based on their relative importance. Empirical analysis on CIFAR-10 dataset verifies the merits of IncReg. Further extensive experiments with popular CNNs on CIFAR-10 and ImageNet datasets show that IncReg achieves comparable to even better results compared with state-of-the-arts. Our source codes and trained models are available here: https://github.com/mingsun-tse/caffe_increg.
BibTeX
@inproceedings{wang2019structured,
  author = {Huan Wang and Qiming Zhang and Yuehai Wang and Roland Hu},
  title = {{Structured Deep Neural Network Pruning by Varying Regularization Parameters}},
  booktitle = {IJCNN},
  year = {2019},
  doi = {10.1109/ijcnn.2019.8852463},
}
Figure