Effective Model Compression via Stage-wise Pruning

Mingyang Zhang, Xinyi Yu, Jingtao Rong, Linlin Ou

TL;DR

Stage-wise Pruning (SWP) addresses a fundamental flaw in AutoML channel pruning: the inaccurate evaluation of subnets caused by insufficient and unfair training within massive weight-sharing supernets. By breaking a deep supernet into independent stages and leveraging inplace distillation alongside balanced channel sampling, SWP ensures highly accurate subnet ranking and delivers state-of-the-art model compression on CIFAR-10 and ImageNet.

What Problem Does This Paper Address?

To avoid the prohibitive computational cost of training thousands of pruned network candidates (subnets) from scratch, modern AutoML pruning methods typically train a single, massive "weight-sharing supernet." Subnets inherit weights directly from this supernet, allowing researchers to quickly evaluate their performance.

However, the authors point out that this approach suffers from two severe issues:

  1. Unfull (Insufficient) Training: Because the search space grows exponentially with network depth, each individual subnet is rarely sampled during training, leading to poor convergence and highly inaccurate performance evaluations.
  2. Unfair Training: Standard random sampling inherently favors lower-indexed channels (the "width" of the layers), leaving higher-indexed channels chronically undertrained.

As a result, the subnets that perform best during the search phase often perform poorly when actually trained from scratch—sometimes failing to even beat simple, uniform pruning.

Figure 1. (a) The expectation of top-1 accuracy collected from ResNet56[10] with different candidate number in one layer. The blue and red dash line denotes ResNet56 trained on CIFAR10[35] under 100 epoches and 500 epoches, respectively. (b) The expectation of top-1 accuracy collected from ResNet18[10] , ResNet34[10] and ResNet56 with different candidate number in one layer. All models are trained under 100 epoches on CIFAR10[35].

Methods

The authors introduce Stage-wise Pruning (SWP) to systematically tackle these training deficiencies through three core components:

  • Stage-wise Supernets: Instead of training a deep supernet all at once, the authors divide it into several independent stages (e.g., 4 stages for ResNet-50 based on feature map downsampling boundaries). This exponentially shrinks the candidate search space within each stage, ensuring every subnet gets adequately trained.
  • Stage-wise Inplace Distillation: To train intermediate stages without external ground-truth labels, the "fullnet" (the largest unpruned network) dynamically transfers its knowledge to smaller subnets within the same iteration. This is achieved by minimizing the $L_2$ (MSE) distance between their stage-wise output feature maps, completely bypassing the need for an expensive, pre-trained teacher model.
Figure 2. Illustration of the stage-wise training. There are three forms of the network, including the fullnet, the subnet and the tinynet. The fullnet infers inputs once to generate and transfer its knowledge to the subnet and the tinynet by minimizing the L2-distance between the their stage-wise output feature maps. It is worthy to note that these three networks are weight sharing.
  • Balanced Sampling (Fullnet + Tinynet): To eliminate width bias, both the fullnet and the "tinynet" (the smallest possible subnet) are explicitly sampled in every training iteration. This ensures every channel is trained at least once per step and bounds the performance of all intermediate subnets.
  • Distributed Evolutionary Algorithm (DEA): A two-tier search strategy. An "EA Manager" optimizes how the total FLOPs budget is distributed across different stages, while independent "Stage EAs" concurrently search for the best subnet configuration within each stage.

Datasets

The method was evaluated on two standard computer vision benchmarks:

  • CIFAR-10: Used for initial validation, analysis of candidate scaling, and baseline pruning comparisons using ResNet-56.
  • ImageNet 2012: Used for large-scale evaluation on MobileNet V1, MobileNet V2, and ResNet-50.

Key Insights and Ideas

  • The Proxy-to-Actual Alignment: The true "aha moment" of the paper is captured in the ranking effectiveness analysis. The authors show that SWP's proxy metric (distillation loss) correlates strongly with actual trained-from-scratch accuracy. In contrast, popular methods like MetaPruning and AutoSlim show almost zero correlation, meaning their search phases are essentially operating at random.
Figure 4. Comparison of ranking effectiveness for Stage-wise Pruning(ours), MetaPruning[27] and AutoSlim[42].
  • Self-Supervision via Inplace Distillation: Traditionally, knowledge distillation requires spending days pre-training a massive teacher network. SWP proves that the fullnet can successfully teach smaller subnets while it is learning itself, achieving identical performance to a static, pre-trained teacher with zero extra training overhead.

Key Results

  • ResNet-56 (CIFAR-10): SWP achieved 95.03% top-1 accuracy with only 61.36M FLOPs, outperforming the unpruned baseline (93.27% @ 125.49M FLOPs) and beating EagleEye (94.66%).
  • ResNet-50 (ImageNet): Under a strict 2.0G FLOPs budget, SWP reached 76.1% top-1 accuracy, outperforming AutoSlim (75.6%) and MetaPruning (75.4%).
  • MobileNet V2 (ImageNet): At 220M FLOPs, SWP achieved 73.4% top-1 accuracy—a 3.6% improvement over the uniform pruning baseline.
  • NAS Integration: When extended to Neural Architecture Search (SWPNas), the method outperformed MobileNetV3-large by 1.7% top-1 accuracy on ImageNet under similar mobile-setting FLOP constraints.

Limitations and Future Work

  • Memory Overhead: Splitting the network into more stages requires saving intermediate feature maps from the fullnet to supervise the subnets. The authors note that increasing the number of stages from 1 to 8 increases the memory footprint from 0.1M to 3M during the search phase.
  • Manual Stage Partitioning: Currently, the optimal number of stages relies on natural downsampling boundaries inherent to architectures like ResNet or MobileNet. Automating or optimizing this stage-partitioning for arbitrary, non-standard network architectures remains an open challenge.

Summary updated

Abstract

Automated Machine Learning(Auto-ML) pruning methods aim at searching a pruning strategy automatically to reduce the computational complexity of deep Convolutional Neural Networks(deep CNNs). However, some previous work found that the results of many Auto-ML pruning methods cannot even surpass the results of the uniformly pruning method. In this paper, the ineffectiveness of Auto-ML pruning which is caused by unfull and unfair training of the supernet is shown. A deep supernet suffers from unfull training because it contains too many candidates. To overcome the unfull training, a stage-wise pruning(SWP) method is proposed, which splits a deep supernet into several stage-wise supernets to reduce the candidate number and utilize inplace distillation to supervise the stage training. Besides, A wide supernet is hit by unfair training since the sampling probability of each channel is unequal. Therefore, the fullnet and the tinynet are sampled in each training iteration to ensure each channel can be overtrained. Remarkably, the proxy performance of the subnets trained with SWP is closer to the actual performance than that of most of the previous Auto-ML pruning work. Experiments show that SWP achieves the state-of-the-art on both CIFAR-10 and ImageNet under the mobile setting.
BibTeX
@article{zhang2023effective,
  author = {Mingyang Zhang and Xinyi Yu and Jingtao Rong and Linlin Ou},
  title = {{Effective Model Compression via Stage-wise Pruning}},
  journal = {Mach. Intell. Res.},
  year = {2023},
  doi = {10.1007/s11633-022-1357-9},
}
Figure