Carrying Out CNN Channel Pruning in a White Box

Yuxin Zhang, Mingbao Lin, Chia-Wen Lin, Jie Chen, Yongjian Wu, Yonghong Tian, Rongrong Ji

Abstract

Most convolutional neural network (CNN) channel pruning methods treat the network as a black box, removing filters based solely on their impact on final output error. This paper introduces "White-Box" pruning, which leverages CNN interpretability to evaluate the contribution of internal channels to specific image categories. By training class-wise masks and preserving channels that benefit the most categories, the method achieves state-of-the-art computation (FLOP) reductions with minimal to no accuracy loss. Experimental results on CIFAR-10 and ImageNet demonstrate that this approach consistently outperforms existing pruning strategies.

1. Introduction

Convolutional Neural Networks (CNNs) are computationally intensive, making them difficult to deploy on resource-constrained edge devices. Channel pruning addresses this by removing entire filters to save compute. However, existing pruning methods—whether based on weight magnitude, sparsity constraints, or AutoML—are fundamentally "black-box" approaches. They optimize for the final classification output while ignoring the internal semantic logic of the network.

Visualization tools reveal that specific channels learn to extract highly specific local features (e.g., one channel might activate for animal heads, while another activates for text). The authors hypothesize that channels useful for recognizing many categories are universally important and should be kept, while channels that only help identify a tiny subset of categories are redundant. This paper proposes a "White-Box" pruning strategy that explicitly measures channel importance through category-driven interpretability.

2. Methodology

To open the black box, the authors propose a pipeline that explicitly measures each channel's contribution to every individual class.

2.1 Class-wise Masks and Soft Labels

The authors introduce a trainable mask for each channel, with a length equal to the number of categories in the dataset. During training, when an image of a specific class is processed, only the corresponding index of the mask is activated to scale the channel's output feature map. To prevent overfitting caused by forcing intermediate layers to predict strict one-hot labels, the authors use a "soft label" approach, smoothing the label distributions using a normal distribution for non-target classes.

The White-Box pruning pipeline: Class-wise masks are trained using soft labels to determine the importance of each channel across different categories.

2.2 Sparsity Regularization and Global Voting

An $L_2$-norm penalty is applied to the masks to encourage sparsity. After a few epochs of training, the absolute sum of a channel's mask values across all classes becomes its "importance score." Channels are ranked globally by this score, and the lowest-scoring channels are iteratively removed until a target FLOP budget is met. Finally, the pruned network is fine-tuned to recover accuracy.

3. Experiments and Results

The White-Box approach was evaluated on CIFAR-10 (using VGG-16, ResNet-56/110, and MobileNet-v2) and ILSVRC-2012 (ImageNet) using ResNet-50.

3.1 Performance Benchmarks

The method consistently outperformed state-of-the-art pruning methods like HRank, GAL, and LFPC. For ResNet-110, it reduced FLOPs by 66.0% while improving top-1 accuracy by 0.62%. For VGG-16, it achieved a 76.4% FLOP reduction with only a negligible 0.45% drop in accuracy.

Comparison of Top-1 Accuracy Change vs. FLOPs Pruning Rate for ResNet-56 and ResNet-110. The White-Box method (red stars) consistently maintains higher accuracy at higher pruning rates compared to other state-of-the-art methods.

On ResNet-50, the method achieved a 45.6% FLOP reduction with a minor 0.83% drop in top-1 accuracy. Pushing the pruning further to a 63.5% FLOP reduction resulted in a 1.94% accuracy drop, translating to a 1.62x actual inference speedup on a V100 GPU.

4. Discussion and Conclusion

This work successfully bridges CNN interpretability with practical network compression. By defining importance through universal applicability across classes, the White-Box approach achieves superior compression-accuracy trade-offs. However, the method is highly sensitive to the soft-label smoothing trick, and its reliance on ground-truth labels limits its current application to supervised classification. Future work could adapt this logic to modern architectures like Vision Transformers or multi-label tasks like object detection.

Summary updated

Abstract

Channel pruning has been long studied to compress convolutional neural networks (CNNs), which significantly reduces the overall computation. Prior works implement channel pruning in an unexplainable manner, which tends to reduce the final classification errors while failing to consider the internal influence of each channel. In this article, we conduct channel pruning in a white box. Through deep visualization of feature maps activated by different channels, we observe that different channels have a varying contribution to different categories in image classification. Inspired by this, we choose to preserve channels contributing to most categories. Specifically, to model the contribution of each channel to differentiating categories, we develop a class-wise mask for each channel, implemented in a dynamic training manner with respect to the input image’s category. On the basis of the learned class-wise mask, we perform a global voting mechanism to remove channels with less category discrimination. Lastly, a fine-tuning process is conducted to recover the performance of the pruned model. To our best knowledge, it is the first time that CNN interpretability theory is considered to guide channel pruning. Extensive experiments on representative image classification tasks demonstrate the superiority of our White-Box over many state-of-the-arts (SOTAs). For instance, on CIFAR-10, it reduces 65.23% floating point operations per seconds (FLOPs) with even 0.62% accuracy improvement for ResNet-110. On ILSVRC-2012, White-Box achieves a 45.6% FLOP reduction with only a small loss of 0.83% in the top-1 accuracy for ResNet-50. Code is available at https://github.com/zyxxmu/White-Box .
BibTeX
@misc{zhang2021carrying,
  author = {Yuxin Zhang and Mingbao Lin and Chia-Wen Lin and Jie Chen and Yongjian Wu and Yonghong Tian and Rongrong Ji},
  title = {{Carrying Out CNN Channel Pruning in a White Box}},
  year = {2021},
  eprint = {2104.11883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.11883},
}
Figure