Abstract
Structural pruning of neural network parameters reduces computation, energy, and memory transfer costs during inference. We propose a novel method that estimates the contribution of a neuron (filter) to the final loss and iteratively removes those with smaller scores. We describe two variations of our method using the first and second-order Taylor expansions to approximate a filter's contribution. Both methods scale consistently across any network layer without requiring per-layer sensitivity analysis and can be applied to any kind of layer, including skip connections. For modern networks trained on ImageNet, we measured experimentally a high (>93%) correlation between the contribution computed by our methods and a reliable estimate of the true importance. Pruning with the proposed methods leads to an improvement over state-of-the-art in terms of accuracy, FLOPs, and parameter reduction. On ResNet-101, we achieve a 40% FLOPS reduction by removing 30% of the parameters, with a loss of 0.02% in the top-1 accuracy on ImageNet.
Introduction
Modern Convolutional Neural Networks (CNNs) are notoriously over-parameterized, making them difficult to deploy on mobile or embedded devices with limited power and memory. Pruning—the process of removing redundant parameters—is the standard solution to this challenge. However, the field has long struggled with a fundamental question: which neurons should be cut?
Many popular methods rely on "magnitude-based pruning," assuming that weights with small absolute values are the least important. This paper argues that this is a flawed assumption; a small weight might be critical to the final output, while a large weight might be redundant. To prune effectively, we need a metric that measures a neuron's actual contribution to the final loss, and we need that metric to be consistent across different layers and complex architectures like those with skip connections.
Methodology
The core idea is to estimate the importance ($I$) of a neuron by calculating how much the loss function would change if that neuron were removed (set to zero). Since re-evaluating the entire network for every single neuron (the "Oracle" approach) is computationally prohibitive for large models, the authors use a Taylor expansion to approximate the change in loss.
- Taylor Expansion Approximation:
- First-Order (Taylor FO): Uses the product of the weight and its gradient: $I \approx (g \cdot w)^2$. This is computationally efficient because gradients are already calculated during standard backpropagation.
- Second-Order (Taylor SO): Incorporates the Hessian (second-order derivatives) for more precision, though it is more memory-intensive.
- Structural Pruning via Gates: Instead of pruning individual weights, the method prunes entire filters by placing "gates" after Batch Normalization (BN) layers. Pruning a gate effectively removes the entire feature map.
- Iterative Pruning & Fine-tuning: The process involves removing a small percentage of neurons (about 2%), fine-tuning the model briefly to recover, and repeating until the target compression is reached.
Key Insights
- Magnitude $\neq$ Importance: Empirical evidence shows that weight magnitude is a poor proxy for importance. The Taylor-based metric shows a >93% correlation with the "true" importance (the Oracle), while magnitude-based metrics lag significantly behind.
- The Power of First-Order: Surprisingly, the first-order Taylor expansion performs nearly as well as the more complex second-order version. This makes the method highly practical for production environments.
- Global Consistency: Because importance is tied to the final loss, scores are comparable across different layers. This eliminates the need for manual "sensitivity analysis" to decide layer-specific pruning rates.
Experimental Results
The method was rigorously tested on standard benchmarks including CIFAR-10 and ImageNet (ILSVRC2012). The results on ImageNet are particularly striking, showing that the proposed Taylor-FO-BN method defines a new state-of-the-art Pareto front for the accuracy-efficiency trade-off.
- ResNet-101: Achieved a 40% reduction in FLOPs and a 30% reduction in parameters with only a 0.02% loss in top-1 accuracy.
- VGG11-BN: Removed 76% of parameters with only a 0.19% loss in accuracy.
- Real-world Speedup: On an NVIDIA Tesla V100, a pruned ResNet-101 showed a 1.59x speedup in inference time for large batches.
Conclusion
By moving beyond simple heuristics to a mathematically grounded importance estimation, this approach allows for significant model compression with negligible accuracy loss. While the current method is "greedy" and makes local decisions at each step, its efficiency and global consistency across layers make it a powerful tool for deploying state-of-the-art models on resource-constrained hardware.
Summary updated