Network Pruning via Performance Maximization

Shangqian Gao, Feihu Huang, Weidong Cai, Heng Huang

Abstract

Traditional channel pruning methods typically rely on minimizing classification loss as a proxy for accuracy. However, a "loss-metric mismatch" often occurs where lower loss does not translate to higher accuracy. We present NPPM (Network Pruning via Performance Maximization), a framework that trains an auxiliary performance predictor to estimate sub-network accuracy directly. By optimizing for this predicted accuracy alongside standard loss, NPPM guides the pruning process toward architectures with superior generalization. Our method achieves state-of-the-art results on CIFAR-10 and ImageNet across various architectures, including ResNet and MobileNetV2.

The Problem: Loss-Metric Mismatch

To deploy deep learning models on resource-constrained devices, channel pruning is used to remove redundant parameters. Most existing methods use training loss (e.g., cross-entropy) to determine which channels to prune. The authors identify a critical flaw: a sub-network with slightly higher loss may actually possess better generalization and higher accuracy than one with lower loss. Because accuracy is a discrete, non-differentiable metric, it cannot be optimized directly via standard gradient descent, leading researchers to rely on loss as an imperfect proxy.

Methodology

NPPM treats accuracy as a first-class citizen by introducing a differentiable proxy for performance. The framework consists of several integrated components:

  1. Differentiable Pruning: The system uses "differentiable gates" (implemented via the Gumbel-Softmax trick) to decide whether a channel is kept or pruned, allowing the architecture itself to be updated through backpropagation.
  2. Performance Prediction Network (PN): A small, stand-alone network using GRUs and fully connected layers is trained to take a sub-network's structure as input and predict its accuracy. This PN serves as the differentiable objective for performance maximization.
  3. Episodic Memory (EM): To train the PN effectively during the pruning process, an episodic memory module stores and re-samples diverse sub-network configurations, preventing the predictor from suffering from catastrophic forgetting or low-diversity data.
Overview of the NPPM framework, illustrating the search for sub-networks, the performance prediction network (PN), and the maximization of predicted accuracy.
  1. Multi-Objective Optimization: The final objective function balances classification loss, the PN's predicted accuracy, and a penalty term designed to reach a specific FLOPs budget.
  2. Gradient Orthogonalization: To ensure the PN provides unique supervision, the authors modify the gradients so that the performance maximization signal is orthogonal to the loss minimization signal.

Experimental Results

NPPM was evaluated on industry-standard benchmarks including CIFAR-10 and ImageNet. The results demonstrate that direct performance maximization consistently outperforms loss-only pruning strategies.

Test accuracy over training epochs. (a) Comparison between NPPM (PM), differentiable gates (DG), and PM without gradient manipulation. (b) Sensitivity analysis of the performance maximization weight lambda.

Key Performance Metrics:

  • ResNet-56 (CIFAR-10): Achieved a +0.36% accuracy increase over the baseline while removing 50% of FLOPs, outperforming methods like AMC and HRank.
  • ResNet-50 (ImageNet): Pruned 56% of FLOPs with only a 0.19% drop in Top-1 accuracy, significantly better than the 0.40% drop seen in competing methods like LeGR.
  • MobileNetV2 (ImageNet): Successfully pruned the model with a +0.02% accuracy gain, a notable result for an already highly optimized architecture.

Key Insights

  • Validation of the Mismatch: Gradient analysis confirms that accuracy-based gradients often point in different directions than loss-based gradients, particularly in the deeper layers of the network. This suggests that performance maximization is most critical for pruning later stages of a CNN.
  • Efficiency: Unlike AutoML-based pruning that requires training thousands of models to learn a predictor, NPPM trains its predictor "in-place" during the pruning trajectory, making it computationally efficient.

Limitations and Future Work

While effective, NPPM introduces new hyperparameters, such as the weight of the performance maximization term ($\lambda$) and the episodic memory size ($K_{max}$), which may require tuning for new architectures. Additionally, the PN relies on mini-batch accuracy, which can be a noisy estimate. Future research could extend this performance maximization philosophy to other compression techniques like quantization or broader neural architecture search (NAS).

Summary updated

Abstract

Channel pruning is a class of powerful methods for model compression. When pruning a neural network, it's ideal to obtain a sub-network with higher accuracy. However, a sub-network does not necessarily have high accuracy with low classification loss (loss-metric mismatch). In the paper, we first consider the loss-metric mismatch problem for pruning and propose a novel channel pruning method for Convolutional Neural Networks (CNNs) by directly maximizing the performance (i.e., accuracy) of sub-networks. Specifically, we train a stand-alone neural network to predict sub-networks' performance and then maximize the output of the network as a proxy of accuracy to guide pruning. Training such a performance prediction network efficiently is not an easy task, and it may potentially suffer from the problem of catastrophic forgetting and the imbalance distribution of sub-networks. To deal with this challenge, we introduce a corresponding episodic memory to update and collect sub-networks during the pruning process. In the experiment section, we further demonstrate that the gradients from the performance prediction network and the classification loss have different directions. Extensive experimental results show that the proposed method can achieve state-of-the-art performance with ResNet, MobileNetV2, and ShuffleNetV2+ on ImageNet and CIFAR-10.
BibTeX
@inproceedings{gao2021network,
  author = {Shangqian Gao and Feihu Huang and Weidong Cai and Heng Huang},
  title = {{Network Pruning via Performance Maximization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00915},
}
Figure