Structural Alignment for Network Pruning through Partial Regularization

Shangqian Gao, Zeyu Zhang, Yanfu Zhang, Feihu Huang, Heng Huang

Abstract

Structural Alignment for Network Pruning (SANP) addresses the performance gap in channel pruning by introducing a partial regularization strategy. By utilizing an Architecture Generator Network (AGN) to identify redundant channels and applying Group Lasso regularization specifically to those weights, SANP enables a model to transition its representational capacity to the target sub-network during training. Experimental results demonstrate that SANP achieves state-of-the-art performance, often improving accuracy while significantly reducing FLOPs across various architectures and datasets.

Introduction

Deep learning models continue to grow in complexity, yet deployment on edge devices like smartphones and IoT sensors remains constrained by hardware limitations. Channel pruning—removing entire filters from Convolutional Neural Networks (CNNs)—is a primary technique for model compression. However, existing methods often suffer from a "structural gap." One-shot pruning typically causes a traumatic drop in performance that requires extensive fine-tuning, while soft pruning methods often over-penalize essential weights, restricting the model's learning capacity. SANP aims to bridge this gap by aligning the full model's weights with the eventual pruned structure during the training process.

Methods

The SANP framework introduces a novel approach to training for sparsity through three integrated components:

  1. Architecture Generator Network (AGN): This auxiliary network, composed of Gated Recurrent Units (GRUs) and dense layers, learns to output a binary mask vector. This mask identifies which channels in the main CNN are redundant and should be targeted for pruning.
  2. Partial Regularization: Unlike standard methods that penalize all weights, SANP applies a Group Lasso penalty exclusively to the channels flagged for removal by the AGN. This encourages "to-be-pruned" weights to shrink toward zero while allowing the "to-be-kept" weights to learn without unnecessary constraints.
  3. Proximal Gradient Optimization: A specialized proximal gradient update allows the model to dynamically adjust which channels are regularized as the AGN evolves, ensuring a smooth transition from the full model to the optimized sub-network.
Overview of the SANP framework showing the interaction between the full model, the sub-network, and the Architecture Generator Network (AGN).

Crucially, partial regularization is initiated after the model has established a solid feature base (typically around 20% into the training process). This "middle-start" strategy prevents premature capacity limitation while ensuring the model is sufficiently aligned before the final pruning step.

Experimental Results

The researchers evaluated SANP on gold-standard benchmarks, including CIFAR-10 and ImageNet (ILSVRC-2012), across architectures such as ResNet and MobileNet-V2.

On CIFAR-10, SANP demonstrated the ability to not only maintain but improve accuracy while significantly reducing computational overhead. For instance, with ResNet-56, the method achieved a 0.32% increase in accuracy while removing 52% of FLOPs. Similar gains were observed on MobileNet-V2, where 46% of FLOPs were pruned with a 0.45% accuracy improvement.

Performance comparison of SANP against other state-of-the-art pruning methods on ResNet-56 and MobileNetV2.

On the large-scale ImageNet dataset, SANP outperformed popular methods like SCOP and DMC. For ResNet-50, it achieved a +0.41% Top-1 accuracy gain over the baseline while pruning 56.2% of the FLOPs. Even on the already-compact MobileNet-V2, SANP trimmed 29.1% of FLOPs while improving Top-1 accuracy by 0.14%.

Discussion and Insights

The success of SANP stems from its "dynamic alignment." Because the AGN is trained alongside the model, the target sub-network adapts as weights evolve, finding more optimal structures than static human-designed heuristics. Furthermore, by avoiding penalties on "winning" channels, the core sub-network can reach higher accuracy than in standard training setups.

While highly effective, future work may address hyperparameter sensitivity regarding the regularization start epoch and strength. Additionally, incorporating direct hardware metrics (such as latency) into the AGN's loss function could further optimize models for specific deployment environments.

Summary updated

Abstract

In this paper, we propose a novel channel pruning method to reduce the computational and storage costs of Convolutional Neural Networks (CNNs). Many existing one-shot pruning methods directly remove redundant structures, which brings a huge gap between the model before and after network pruning. This gap will no doubt result in performance loss for network pruning. To mitigate this gap, we first learn a target sub-network during the model training process, and then we use this sub-network to guide the learning of model weights through partial regularization. The target sub-network is learned and produced by using an architecture generator, and it can be optimized efficiently. In addition, we also derive the proximal gradient for our proposed partial regularization to facilitate the structural alignment process. With these designs, the gap between the pruned model and the sub-network is reduced, thus improving the pruning performance. Empirical results also suggest that the sub-network found by our method has a much higher performance than the one-shot pruning setting. Extensive experiments show that our method can achieve state-of-the-art performances on CIFAR-10 and ImageNet with ResNets and MobileNet-V2.
BibTeX
@inproceedings{gao2023structural,
  author = {Shangqian Gao and Zeyu Zhang and Yanfu Zhang and Feihu Huang and Heng Huang},
  title = {{Structural Alignment for Network Pruning through Partial Regularization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01596},
}
Figure