Abstract
Deep Convolutional Neural Networks (CNNs) are powerful but computationally expensive, hindering their deployment on resource-constrained mobile devices. While AutoML-based channel pruning has emerged as a promising solution, existing methods often rely on "supernets" that suffer from "unfull" and "unfair" training. This paper introduces Stage-wise Pruning (SWP), a novel framework that decomposes the pruning process into independent stages. By utilizing "inplace distillation" and a distributed evolutionary search, SWP ensures that all candidate subnets are fully trained and accurately evaluated. Our method achieves state-of-the-art results on ImageNet and CIFAR-10 across multiple architectures, including ResNet and MobileNet.
Introduction
Channel pruning—the removal of redundant neurons—is a standard technique for compressing CNNs. Recent advancements have shifted toward using AutoML to find optimal pruning configurations. However, these methods typically train a single massive supernet containing all possible pruned subnets. This approach faces two critical challenges:
- Unfull Training: The astronomical number of possible subnets (e.g., $32^{10}$) means any individual subnet is rarely sampled, leading to poor convergence and inaccurate performance rankings.
- Unfair Training: Weight-sharing schemes often bias training toward channels with smaller indices, leaving the rest of the network under-optimized.
Methods
The proposed Stage-wise Pruning (SWP) addresses these flaws through a "divide and conquer" strategy.
Stage-wise Splitting and Inplace Distillation
Instead of a global supernet, the network is divided into $N$ independent stages. This exponentially reduces the search space from $g^L$ to $g^{L/N}$, where $g$ is the number of candidates per layer and $L$ is the number of layers. To provide supervision for these intermediate stages, the authors employ inplace distillation. In each training iteration, the "fullnet" (unpruned model) acts as a teacher for a sampled "subnet" and the "tinynet" (smallest possible model). Feature maps from the fullnet supervise the others via Mean Squared Error (MSE) loss, ensuring architectural compatibility and robust training.
Distributed Evolutionary Algorithm (DEA)
To optimize the configuration, SWP uses a two-level evolutionary approach. An EA Manager determines the computational budget (FLOPs) for each stage, while individual Stage-wise EAs find the best channel configurations within those specific budgets.
Key Insights: Solving the Evaluation Gap
A critical contribution of SWP is improving the correlation between proxy metrics used during search and the actual final performance of the pruned model. In traditional methods, the proxy accuracy often fails to reflect the true potential of a subnet due to insufficient training. By overtraining the extremes (fullnet and tinynet) and reducing the search space, SWP ensures that the distillation loss serves as a highly reliable proxy for final top-1 accuracy.
Experimental Results
SWP was evaluated on CIFAR-10 and ImageNet 2012 using ResNet and MobileNet backbones:
- ResNet-50 (ImageNet): At a 2.0G FLOP budget, SWP achieved 76.1% Top-1 accuracy, surpassing the baseline (74.8%) and AutoSlim (75.6%).
- MobileNet V2: In extreme compression scenarios (43M FLOPs), SWP maintained 60.7% accuracy, significantly outperforming MetaPruning (58.3%).
- NAS Integration: When combined with Neural Architecture Search, the resulting "SWPNas" model reached 76.9% accuracy on ImageNet with only 214M FLOPs.
Conclusion
Stage-wise Pruning effectively mitigates the training biases inherent in supernet-based AutoML. By simplifying the search space and leveraging inplace distillation, SWP provides a more accurate and efficient path for model compression. Future work may extend this stage-wise paradigm to complex tasks like object detection and semantic segmentation.
Summary updated