Abstract
Traditional channel pruning methods typically rely on minimizing classification loss as a proxy for accuracy. However, a "loss-metric mismatch" often occurs where lower loss does not translate to higher accuracy. We present NPPM (Network Pruning via Performance Maximization), a framework that trains an auxiliary performance predictor to estimate sub-network accuracy directly. By optimizing for this predicted accuracy alongside standard loss, NPPM guides the pruning process toward architectures with superior generalization. Our method achieves state-of-the-art results on CIFAR-10 and ImageNet across various architectures, including ResNet and MobileNetV2.
The Problem: Loss-Metric Mismatch
To deploy deep learning models on resource-constrained devices, channel pruning is used to remove redundant parameters. Most existing methods use training loss (e.g., cross-entropy) to determine which channels to prune. The authors identify a critical flaw: a sub-network with slightly higher loss may actually possess better generalization and higher accuracy than one with lower loss. Because accuracy is a discrete, non-differentiable metric, it cannot be optimized directly via standard gradient descent, leading researchers to rely on loss as an imperfect proxy.
Methodology
NPPM treats accuracy as a first-class citizen by introducing a differentiable proxy for performance. The framework consists of several integrated components:
- Differentiable Pruning: The system uses "differentiable gates" (implemented via the Gumbel-Softmax trick) to decide whether a channel is kept or pruned, allowing the architecture itself to be updated through backpropagation.
- Performance Prediction Network (PN): A small, stand-alone network using GRUs and fully connected layers is trained to take a sub-network's structure as input and predict its accuracy. This PN serves as the differentiable objective for performance maximization.
- Episodic Memory (EM): To train the PN effectively during the pruning process, an episodic memory module stores and re-samples diverse sub-network configurations, preventing the predictor from suffering from catastrophic forgetting or low-diversity data.
- Multi-Objective Optimization: The final objective function balances classification loss, the PN's predicted accuracy, and a penalty term designed to reach a specific FLOPs budget.
- Gradient Orthogonalization: To ensure the PN provides unique supervision, the authors modify the gradients so that the performance maximization signal is orthogonal to the loss minimization signal.
Experimental Results
NPPM was evaluated on industry-standard benchmarks including CIFAR-10 and ImageNet. The results demonstrate that direct performance maximization consistently outperforms loss-only pruning strategies.
Key Performance Metrics:
- ResNet-56 (CIFAR-10): Achieved a +0.36% accuracy increase over the baseline while removing 50% of FLOPs, outperforming methods like AMC and HRank.
- ResNet-50 (ImageNet): Pruned 56% of FLOPs with only a 0.19% drop in Top-1 accuracy, significantly better than the 0.40% drop seen in competing methods like LeGR.
- MobileNetV2 (ImageNet): Successfully pruned the model with a +0.02% accuracy gain, a notable result for an already highly optimized architecture.
Key Insights
- Validation of the Mismatch: Gradient analysis confirms that accuracy-based gradients often point in different directions than loss-based gradients, particularly in the deeper layers of the network. This suggests that performance maximization is most critical for pruning later stages of a CNN.
- Efficiency: Unlike AutoML-based pruning that requires training thousands of models to learn a predictor, NPPM trains its predictor "in-place" during the pruning trajectory, making it computationally efficient.
Limitations and Future Work
While effective, NPPM introduces new hyperparameters, such as the weight of the performance maximization term ($\lambda$) and the episodic memory size ($K_{max}$), which may require tuning for new architectures. Additionally, the PN relies on mini-batch accuracy, which can be a noisy estimate. Future research could extend this performance maximization philosophy to other compression techniques like quantization or broader neural architecture search (NAS).
Summary updated