Abstract
This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet. The code is available at https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020.
Introduction
Deep learning models are notoriously "heavy," making them difficult to deploy on mobile phones or edge devices. Pruning—the process of removing redundant parameters or filters—is the standard solution. However, most existing pruning methods rely on heuristics, such as removing filters with the smallest weights, which can be unreliable. The authors argue that these heuristics are often "noisy"; a filter might have large weights or high activation not because it is useful for the task, but due to internal correlations or data fluctuations. SCOP (Scientific Control for Reliable Pruning) addresses this by introducing a "scientific control" to prove that a specific filter actually contributes to the model's predictive power.
Methodology: Pruning via Scientific Control
The core innovation of SCOP is the application of the Knockoff Method to deep learning, treating neural network pruning like a clinical trial.
Generating Knockoffs
For every real feature map produced by a layer, the authors generate a "knockoff" feature. This knockoff mimics the statistical distribution of the real data but is mathematically constructed to be independent of the ground-truth labels. This acts as a "placebo" feature, allowing the model to distinguish between useful signals and random noise.
Adversarial Selection
The framework inserts an adversarial selection layer that assigns learnable scaling factors: $\beta$ for real features and $\tilde{\beta}$ for knockoffs. These factors are constrained such that $\beta + \tilde{\beta} = 1$.
During a brief training phase, the network learns which features to prioritize. If a filter is truly important, its real feature should "outperform" its knockoff counterpart, resulting in a high $\beta$. If the knockoff (which has no label information) is just as useful as the real feature, the filter is deemed redundant.
Pruning Strategy
Filters are ranked by the difference $I = \beta - \tilde{\beta}$. Filters that fail to significantly outperform their "placebo" counterparts are pruned. The paper also provides a theoretical proof showing that the "knockoff" property (being label-independent but distributionally similar) can be approximately preserved as data flows through the nonlinear layers of a CNN, ensuring efficiency in generation.
Experimental Results
SCOP was evaluated on standard computer vision benchmarks, CIFAR-10 and ImageNet (ILSVRC-2012), consistently outperforming state-of-the-art (SOTA) methods like HRank and FPGM.
- ResNet-101 on ImageNet: SCOP reduced parameters by 57.8% and FLOPs by 60.2% with a negligible top-1 accuracy loss of only 0.01%.
- ResNet-50 on ImageNet: It achieved a 45.3% FLOPs reduction with only a 0.20% increase in top-1 error, significantly better than previous methods that often lost 1-2% accuracy at similar compression rates.
- MobileNetV2: Even on models already designed to be "lightweight," SCOP managed to prune 40.3% of FLOPs with only a 0.24% accuracy drop.
Conclusion
By reframing pruning as a feature selection problem with a rigorous control group, SCOP isolates the specific association between a filter and the final prediction. This method allows for massive reductions in model size with virtually no loss in accuracy. The authors suggest that this concept of scientific control could be extended to Neural Architecture Search (NAS), helping to design more efficient networks from scratch.
Summary updated