TL;DR
Stage-wise Pruning (SWP) addresses a fundamental flaw in AutoML channel pruning: the inaccurate evaluation of subnets caused by insufficient and unfair training within massive weight-sharing supernets. By breaking a deep supernet into independent stages and leveraging inplace distillation alongside balanced channel sampling, SWP ensures highly accurate subnet ranking and delivers state-of-the-art model compression on CIFAR-10 and ImageNet.
What Problem Does This Paper Address?
To avoid the prohibitive computational cost of training thousands of pruned network candidates (subnets) from scratch, modern AutoML pruning methods typically train a single, massive "weight-sharing supernet." Subnets inherit weights directly from this supernet, allowing researchers to quickly evaluate their performance.
However, the authors point out that this approach suffers from two severe issues:
- Unfull (Insufficient) Training: Because the search space grows exponentially with network depth, each individual subnet is rarely sampled during training, leading to poor convergence and highly inaccurate performance evaluations.
- Unfair Training: Standard random sampling inherently favors lower-indexed channels (the "width" of the layers), leaving higher-indexed channels chronically undertrained.
As a result, the subnets that perform best during the search phase often perform poorly when actually trained from scratch—sometimes failing to even beat simple, uniform pruning.
Methods
The authors introduce Stage-wise Pruning (SWP) to systematically tackle these training deficiencies through three core components:
- Stage-wise Supernets: Instead of training a deep supernet all at once, the authors divide it into several independent stages (e.g., 4 stages for ResNet-50 based on feature map downsampling boundaries). This exponentially shrinks the candidate search space within each stage, ensuring every subnet gets adequately trained.
- Stage-wise Inplace Distillation: To train intermediate stages without external ground-truth labels, the "fullnet" (the largest unpruned network) dynamically transfers its knowledge to smaller subnets within the same iteration. This is achieved by minimizing the $L_2$ (MSE) distance between their stage-wise output feature maps, completely bypassing the need for an expensive, pre-trained teacher model.
- Balanced Sampling (Fullnet + Tinynet): To eliminate width bias, both the fullnet and the "tinynet" (the smallest possible subnet) are explicitly sampled in every training iteration. This ensures every channel is trained at least once per step and bounds the performance of all intermediate subnets.
- Distributed Evolutionary Algorithm (DEA): A two-tier search strategy. An "EA Manager" optimizes how the total FLOPs budget is distributed across different stages, while independent "Stage EAs" concurrently search for the best subnet configuration within each stage.
Datasets
The method was evaluated on two standard computer vision benchmarks:
- CIFAR-10: Used for initial validation, analysis of candidate scaling, and baseline pruning comparisons using ResNet-56.
- ImageNet 2012: Used for large-scale evaluation on MobileNet V1, MobileNet V2, and ResNet-50.
Key Insights and Ideas
- The Proxy-to-Actual Alignment: The true "aha moment" of the paper is captured in the ranking effectiveness analysis. The authors show that SWP's proxy metric (distillation loss) correlates strongly with actual trained-from-scratch accuracy. In contrast, popular methods like MetaPruning and AutoSlim show almost zero correlation, meaning their search phases are essentially operating at random.
- Self-Supervision via Inplace Distillation: Traditionally, knowledge distillation requires spending days pre-training a massive teacher network. SWP proves that the fullnet can successfully teach smaller subnets while it is learning itself, achieving identical performance to a static, pre-trained teacher with zero extra training overhead.
Key Results
- ResNet-56 (CIFAR-10): SWP achieved 95.03% top-1 accuracy with only 61.36M FLOPs, outperforming the unpruned baseline (93.27% @ 125.49M FLOPs) and beating EagleEye (94.66%).
- ResNet-50 (ImageNet): Under a strict 2.0G FLOPs budget, SWP reached 76.1% top-1 accuracy, outperforming AutoSlim (75.6%) and MetaPruning (75.4%).
- MobileNet V2 (ImageNet): At 220M FLOPs, SWP achieved 73.4% top-1 accuracy—a 3.6% improvement over the uniform pruning baseline.
- NAS Integration: When extended to Neural Architecture Search (SWPNas), the method outperformed MobileNetV3-large by 1.7% top-1 accuracy on ImageNet under similar mobile-setting FLOP constraints.
Limitations and Future Work
- Memory Overhead: Splitting the network into more stages requires saving intermediate feature maps from the fullnet to supervise the subnets. The authors note that increasing the number of stages from 1 to 8 increases the memory footprint from 0.1M to 3M during the search phase.
- Manual Stage Partitioning: Currently, the optimal number of stages relies on natural downsampling boundaries inherent to architectures like ResNet or MobileNet. Automating or optimizing this stage-partitioning for arbitrary, non-standard network architectures remains an open challenge.
Summary updated