SCOP: Scientific Control for Reliable Neural Network Pruning

Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, Chang Xu

Introduction

Modern Convolutional Neural Networks (CNNs) are notoriously over-parameterized, making them computationally heavy and difficult to deploy on edge devices like smartphones or wearables. While filter pruning—identifying and removing entire convolutional filters to create a smaller, faster network—is a standard solution, most existing methods rely on shaky heuristics.

SCOP (Scientific Control for Reliable Neural Network Pruning) introduces a rigorous statistical framework to model compression. By using "knockoff" features as a scientific control group, the algorithm identifies and removes redundant filters that do not actually contribute to label prediction. This approach achieves massive FLOP reductions (over 60% on ResNet-101) with virtually no loss in accuracy.

The Challenge of Heuristic Pruning

Traditional pruning often assumes that filters with small weights (norms) are less important. However, this magnitude-based assumption is flawed. A filter might have a large norm simply because it is reacting to noise or redundant correlations in the data. Without a baseline to compare against, it is difficult to determine if a filter is providing a genuinely useful signal or just "hallucinating" importance.

Methodology: A "Clinical Trial" for Networks

To address the lack of a reliable baseline, SCOP treats pruning like a scientific experiment, introducing a control group to evaluate filter importance rigorously.

The SCOP pipeline: Real and knockoff data are fed into the network, where learnable scaling factors compete to identify and prune redundant filters.

The SCOP framework operates through four key steps:

  1. The Control Group (Knockoffs): For every real feature map, SCOP generates a "knockoff" counterpart. These knockoffs share the same statistical distribution as the real data but are fundamentally independent of the actual labels—acting as a "placebo."
  2. Information Propagation: Instead of training a separate generative model for every single layer, the authors mathematically prove that if you start with knockoff input data, the "knockoff property" is approximately preserved as the data flows through the network's layers.
  3. Adversarial Selection: A special layer is inserted that mixes real features and knockoff features using learnable scaling factors: $\beta$ for real features and $\tilde{\beta}$ for knockoff features.
  4. The Pruning Metric: During a brief training phase, the network "competes" to use either the real or knockoff features. If a filter's knockoff scale ($\tilde{\beta}$) is large compared to its real scale ($\beta$), the filter is essentially processing noise or irrelevant patterns. These filters are marked as redundant and pruned.

Experimental Results

The authors evaluated SCOP on standard computer vision benchmarks, including CIFAR-10 and the large-scale ImageNet (ILSVRC-2012) dataset. SCOP consistently outperformed state-of-the-art methods like HRank and FPGM across various architectures, including ResNet-20, 34, 50, 101, and MobileNetV2.

  • ResNet-101 on ImageNet: SCOP reduced parameters by 57.8% and computational cost (FLOPs) by 60.2%, while suffering a negligible 0.01% drop in top-1 accuracy.
  • ResNet-50 on ImageNet: The method achieved a 45.3% FLOP reduction with only a 0.20% accuracy loss, significantly outperforming previous methods that often see drops of 1% or more at similar compression levels.
  • Real-world Speedup: On an NVIDIA V100 GPU, the pruned ResNet-101 demonstrated a 50.6% realistic acceleration, proving that theoretical FLOP savings translate directly to actual performance gains.

Limitations and Future Directions

While highly effective, SCOP does have some limitations. Generating the initial input knockoffs requires a high-quality generative model (like a GAN), which adds complexity to the pipeline, even though this data only needs to be generated once. Additionally, the preservation of the knockoff property through linear layers is technically "approximate," though the authors found strict enforcement wasn't always necessary in practice.

Looking forward, the authors suggest that this "scientific control" philosophy could be extended beyond pruning. Areas like Neural Architecture Search (NAS) could benefit from similar frameworks to determine if complex architectural components are truly necessary or simply overfitting to specific data patterns.

Summary updated

Abstract

This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet. The code is available at https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020.
BibTeX
@inproceedings{tang2020scop,
  author = {Yehui Tang and Yunhe Wang and Yixing Xu and Dacheng Tao and Chunjing Xu and Chao Xu and Chang Xu},
  title = {{SCOP: Scientific Control for Reliable Neural Network Pruning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure