Abstract
Parameter pruning is a critical technique for deploying deep Convolutional Neural Networks (CNNs) on resource-constrained devices. While regularization-based pruning is effective, traditional methods often apply large, static penalties that can destabilize a network's fragile expressiveness. We introduce IncReg (Incremental Regularization), a novel structured pruning method that gradually increases penalties on unimportant weights. By allowing the network to adapt incrementally, IncReg achieves significant compression and acceleration with minimal accuracy loss. Our evaluations on ImageNet using VGG-16 and ResNet-50 demonstrate that IncReg outperforms state-of-the-art methods, particularly at high speedup ratios.
Introduction
The deployment of modern CNNs is often hindered by their immense computational and memory requirements. Pruning addresses this by removing redundant parameters. However, existing regularization-based methods typically use a constant regularization factor (e.g., Lasso) to drive weights toward zero. This "one-size-fits-all" approach neglects the fact that different weight groups have varying levels of importance. Applying a massive penalty abruptly can cause the model's performance to collapse before it can redistribute the representational burden to the remaining parameters.
The IncReg Method
IncReg treats pruning as a dynamic, adaptive process. Instead of a fixed penalty, the regularization factor for each weight group is updated at every iteration based on its relative importance.
Incremental Penalty and Punishment Function
The core of IncReg is a piece-wise linear punishment function that determines the change in the penalty factor ($\Delta\lambda_g$) for each weight group $g$. Weights are ranked by their importance, typically determined by an averaged $L_1$-norm over time to ensure stability.
As shown in the figure above, weight groups identified as "unimportant" receive an increasing penalty, while "important" groups receive a negative increment (a reward). This protects vital parameters from regularization noise and encourages the network to gently transfer knowledge to the most significant weights.
Structured Sparsity and Removal
IncReg focuses on structured sparsity—removing entire rows or columns—which translates directly to hardware acceleration. Once a weight group's magnitude falls below a threshold (e.g., $10^{-6}$), it is permanently removed. This process continues until the target pruning ratio for the layer is achieved.
Experimental Results
We evaluated IncReg on the ImageNet dataset using VGG-16 and ResNet-50 architectures, comparing it against several state-of-the-art methods including SSL, AFP, CP, and SPP.
Performance on VGG-16
IncReg demonstrates remarkable robustness at high compression rates. For VGG-16, at a 4x speedup, IncReg saw only a 0.8% increase in top-5 error. At a 5x speedup, the error increase remained manageable at 1.5%, whereas competing methods like FP saw errors spike by over 14%.
Performance on ResNet-50
On the more complex ResNet-50 architecture, IncReg achieved a 2x speedup with a remarkably low 0.1% error increase, significantly outperforming CP (1.4%) and SPP (0.8%). Real-world testing on a Titan 1080Ti confirmed these theoretical gains, with a pruned VGG-16 model achieving a 2.6x actual reduction in GPU inference time.
Conclusion
IncReg provides a more sophisticated, "gentle" approach to network pruning. By using incremental regularization based on relative importance, it preserves the model's expressiveness during the compression process. The method's ability to maintain high accuracy at extreme speedup ratios makes it a powerful tool for optimizing CNNs for edge and mobile deployment. Future work may explore automated pruning ratio determination to further enhance its efficiency.
Summary updated