Abstract
Structural Alignment for Network Pruning (SANP) addresses the performance gap in channel pruning by introducing a partial regularization strategy. By utilizing an Architecture Generator Network (AGN) to identify redundant channels and applying Group Lasso regularization specifically to those weights, SANP enables a model to transition its representational capacity to the target sub-network during training. Experimental results demonstrate that SANP achieves state-of-the-art performance, often improving accuracy while significantly reducing FLOPs across various architectures and datasets.
Introduction
Deep learning models continue to grow in complexity, yet deployment on edge devices like smartphones and IoT sensors remains constrained by hardware limitations. Channel pruning—removing entire filters from Convolutional Neural Networks (CNNs)—is a primary technique for model compression. However, existing methods often suffer from a "structural gap." One-shot pruning typically causes a traumatic drop in performance that requires extensive fine-tuning, while soft pruning methods often over-penalize essential weights, restricting the model's learning capacity. SANP aims to bridge this gap by aligning the full model's weights with the eventual pruned structure during the training process.
Methods
The SANP framework introduces a novel approach to training for sparsity through three integrated components:
- Architecture Generator Network (AGN): This auxiliary network, composed of Gated Recurrent Units (GRUs) and dense layers, learns to output a binary mask vector. This mask identifies which channels in the main CNN are redundant and should be targeted for pruning.
- Partial Regularization: Unlike standard methods that penalize all weights, SANP applies a Group Lasso penalty exclusively to the channels flagged for removal by the AGN. This encourages "to-be-pruned" weights to shrink toward zero while allowing the "to-be-kept" weights to learn without unnecessary constraints.
- Proximal Gradient Optimization: A specialized proximal gradient update allows the model to dynamically adjust which channels are regularized as the AGN evolves, ensuring a smooth transition from the full model to the optimized sub-network.
Crucially, partial regularization is initiated after the model has established a solid feature base (typically around 20% into the training process). This "middle-start" strategy prevents premature capacity limitation while ensuring the model is sufficiently aligned before the final pruning step.
Experimental Results
The researchers evaluated SANP on gold-standard benchmarks, including CIFAR-10 and ImageNet (ILSVRC-2012), across architectures such as ResNet and MobileNet-V2.
On CIFAR-10, SANP demonstrated the ability to not only maintain but improve accuracy while significantly reducing computational overhead. For instance, with ResNet-56, the method achieved a 0.32% increase in accuracy while removing 52% of FLOPs. Similar gains were observed on MobileNet-V2, where 46% of FLOPs were pruned with a 0.45% accuracy improvement.
On the large-scale ImageNet dataset, SANP outperformed popular methods like SCOP and DMC. For ResNet-50, it achieved a +0.41% Top-1 accuracy gain over the baseline while pruning 56.2% of the FLOPs. Even on the already-compact MobileNet-V2, SANP trimmed 29.1% of FLOPs while improving Top-1 accuracy by 0.14%.
Discussion and Insights
The success of SANP stems from its "dynamic alignment." Because the AGN is trained alongside the model, the target sub-network adapts as weights evolve, finding more optimal structures than static human-designed heuristics. Furthermore, by avoiding penalties on "winning" channels, the core sub-network can reach higher accuracy than in standard training setups.
While highly effective, future work may address hyperparameter sensitivity regarding the regularization start epoch and strength. Additionally, incorporating direct hardware metrics (such as latency) into the AGN's loss function could further optimize models for specific deployment environments.
Summary updated