Abstract
Deep network pruning is an effective method to reduce the storage and computation cost of deep neural networks when applying them to resource-limited devices. Among many pruning granularities, neuron level pruning will remove redundant neurons and filters in the model and result in thinner networks. In this paper, we propose a gradually global pruning scheme for neuron level pruning. In each pruning step, a small percent of neurons were selected and dropped across all layers in the model. We also propose a simple method to eliminate the biases in evaluating the importance of neurons to make the scheme feasible. Compared with layer-wise pruning scheme, our scheme avoid the difficulty in determining the redundancy in each layer and is more effective for deep networks. Our scheme would automatically find a thinner sub-network in original network under a given performance.
Introduction
Deep learning models like VGG and AlexNet are notoriously over-parameterized, often containing millions or billions of parameters. While this redundancy aids in training stability and convergence, it poses significant challenges for deployment on resource-constrained devices such as smartphones or IoT sensors due to high memory and computational overhead.
Current pruning techniques generally fall into two categories: connection-level pruning and layer-wise neuron pruning. Connection-level pruning creates sparse weight matrices which, while theoretically efficient, often fail to provide real-world speedups on standard hardware not optimized for sparsity. Layer-wise neuron pruning removes entire filters to create "thinner" networks but requires manual determination of redundancy for each layer—a tedious and often sub-optimal process. This paper proposes Gradually Global Pruning (GGP) to automate this process by ranking and removing neurons globally across the entire network.
Methodology
Global Scoring and Bias Correction
To implement a global pruning strategy, a unified metric is required to compare neuron importance across different layers. The authors evaluated three metrics:
- Mean Response ($\bar{R}$): The average output of a neuron.
- Standard Deviation of Response ($\sigma(R)$): A measure of output variance across inputs.
- Average Absolute Weight Sum (AAWS): The average magnitude of weights connected to a neuron.
A significant challenge identified was positional bias, where neurons in earlier layers naturally exhibit higher raw scores than those in deeper layers. To address this, the authors introduced a Modified Contribution Score. By dividing each neuron's score by the average score of its parent layer, the distribution is "flattened," enabling a fair global ranking across the entire architecture.
The Gradually Global Pruning Scheme
Rather than aggressive one-shot pruning, GGP employs a "select-prune-fine-tune" loop. In each iteration, a small pruning ratio ($r \approx 5\%$) of the least important neurons is removed globally. This is followed by a brief fine-tuning phase to allow the remaining network to compensate for the loss. This gradual approach ensures the model remains near a local optimum, making the recovery process stable and efficient compared to more aggressive methods.
Results and Discussion
The researchers evaluated GGP using a VGG-like architecture on the CIFAR-10 dataset.
- Metric Performance: AAWS emerged as the most effective metric. Notably, AAWS is data-independent, allowing for score calculation directly from model weights without requiring a validation dataset.
- Efficiency and Accuracy: The method successfully reduced a VGG-like network from 4,736 neurons to 3,310. The resulting model maintained an accuracy of 86.76%, representing only a marginal decrease from the original 87.32%.
- Global vs. Local: GGP naturally identifies layer redundancy. Experiments showed the algorithm pruned higher-level layers more aggressively than low-level feature extractors, a structural optimization that is difficult to achieve with manual layer-wise pruning.
Conclusion
Gradually Global Pruning provides a robust framework for automatically discovering thinner sub-networks. By eliminating positional bias and employing an iterative pruning schedule, GGP outperforms traditional proportional layer-wise strategies. Future work will involve extending this methodology to larger datasets like ImageNet and more complex architectures such as ResNets.
Summary updated