Improving Network Slimming With Nonconvex Regularization

Kevin Bui, Fredrick Park, Shuai Zhang, Yingyong Qi, Jack XinIEEE Access 2021 · DOI

TL;DR

This paper proposes using nonconvex regularization techniques to improve network slimming, a method for compressing convolutional neural networks (CNNs) by pruning channels. By replacing the standard L1 regularization with nonconvex penalties like L_p, Tl1, MCP, and SCAD, the authors demonstrate that it's possible to achieve more aggressive compression with comparable or even better accuracy, especially after retraining.

What Problem Does This Paper Address?

Modern deep neural networks, particularly CNNs, have achieved remarkable success in various computer vision tasks. However, their sheer size and computational demands make them unsuitable for deployment on resource-constrained edge devices like smartphones and drones. Network slimming is a popular technique to address this by identifying and removing redundant channels during training. The standard approach uses L1 regularization on channel-associated scaling factors, which encourages sparsity. While effective, L1 regularization has limitations: it may not always yield the sparsest solution and can be sensitive to outliers. This paper aims to improve upon network slimming by exploring alternative, nonconvex regularization methods that can potentially lead to more efficient and accurate compressed models.

Methods

The core of this work is the replacement of the L1 regularization penalty in network slimming with several nonconvex, sparsity-inducing penalties. The authors investigate:

  • L_p penalty (0 < p < 1): This penalty interpolates between L0 and L1 and has been shown to yield sparser solutions than L1 in some contexts.
  • Transformed L1 (Tl1) penalty: This method is designed to be unbiased and has shown good performance in compressed sensing and deep learning.
  • Minimax Concave Penalty (MCP): This penalty aims to provide unbiased estimates and has been used for variable selection.
  • Smoothly Clipped Absolute Deviation (SCAD) penalty: Similar to MCP, SCAD is designed to be unbiased and promote sparsity.

These nonconvex regularizers are integrated into the network slimming framework by applying them to the scaling factors within the batch normalization layers. The training process utilizes stochastic gradient descent for the main loss and subgradient descent for the regularization terms. After training, channels with small scaling factors are pruned, and the remaining model is often retrained to recover accuracy.

Datasets

The authors evaluate their proposed methods on the following standard image classification datasets:

  • CIFAR-10: A dataset with 50,000 training images and 10,000 test images across 10 classes.
  • CIFAR-100: Similar to CIFAR-10 but with 100 classes.
  • SVHN (Street View House Numbers): A dataset of 32x32 color images of house numbers.

Key Insights and Ideas

  • Nonconvexity for Better Sparsity: The paper's central idea is that nonconvex regularizers, by having contour plots that bend more towards the axes, can encourage solutions to lie on the corners, leading to sparser solutions compared to L1.
  • Robustness to Pruning: Certain nonconvex regularizers, particularly Tl1 with a smaller alpha parameter, demonstrate greater robustness to channel pruning, meaning the model's accuracy degrades less severely as more channels are removed.
  • Trade-offs in Compression and Accuracy: While some nonconvex methods (like L_p) achieve higher compression ratios (more parameters/FLOPs pruned), they might lead to a slight drop in test accuracy before retraining. Others (like MCP and SCAD) offer comparable compression to L1 but with better test accuracies after retraining.
  • Scaling Factor Distribution Matters: Analysis of scaling factor distributions reveals that regularizers promoting more scaling factors to be very close to zero (like Tl1 with small alpha) lead to better robustness against pruning.
  • Retraining is Crucial: For aggressive pruning, retraining the pruned model is essential to recover lost accuracy, and in many cases, the retrained models can even surpass the original baseline accuracy.

Key Results

The paper reports several key findings across different network architectures (VGG-19, DenseNet-40, ResNet-164) and datasets:

  • Improved Compression: Nonconvex regularizers, especially L_p and Tl1, generally achieve higher parameter and FLOP reduction percentages compared to L1, particularly at higher channel pruning ratios.
  • Accuracy Preservation: Tl1, especially with a smaller alpha parameter (e.g., a=0.5), shows a remarkable ability to preserve model accuracy even when a significant portion of channels are pruned.
  • Competitive Performance: MCP and SCAD often achieve better test accuracies after retraining compared to L1, with similar compression levels.
  • Robustness: Figures 3, 4, and 5 illustrate that models trained with Tl1, L_p (for p=1/2, 3/4), MCP, and SCAD are generally more robust to channel pruning than L1, showing less accuracy degradation.
  • Retraining Benefits: Retraining the pruned models consistently improved test accuracies, often exceeding the original baseline accuracy, especially for VGG-19 and DenseNet-40.

Limitations and Future Work

  • Hyperparameter Tuning: The performance of nonconvex regularizers can be sensitive to hyperparameter choices (e.g., alpha for Tl1, MCP, SCAD, and p for L_p), requiring careful tuning.
  • Computational Cost: While aiming for efficiency, the training process with nonconvex regularizers might involve subgradient computations, which can add to the overall training time.
  • Generalization to Other Architectures: The study focuses on specific CNN architectures. Further investigation is needed to confirm the effectiveness of these methods on a wider range of modern architectures.

The authors suggest several promising future directions:

  • Optimization Algorithms: Developing optimization algorithms based on relaxed variable splitting methods for other nonconvex regularizers.
  • Normalization Layers: Extending nonconvex network slimming to other normalization techniques like layer normalization and group normalization.
  • Compact CNNs: Adapting these methods to state-of-the-art compact CNN architectures such as MobileNetV2 and ShuffleNet.

Summary updated

BibTeX
@article{bui2021improving,
  author = {Kevin Bui and Fredrick Park and Shuai Zhang and Yingyong Qi and Jack Xin},
  title = {{Improving Network Slimming With Nonconvex Regularization}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3105366},
}
Figure