Towards thinner convolutional neural networks through gradually global pruning

Zhengtao Wang, Ce Zhu, Zhiqiang Xia, Qi Guo, Yipeng Liu

TL;DR

This paper introduces Gradually Global Pruning (GGP), a method that thins out Convolutional Neural Networks (CNNs) by removing redundant neurons across the entire model simultaneously rather than layer-by-layer. By using a simple normalization technique to compare neuron importance across different layers, the authors achieve significant model size reduction while maintaining high accuracy, outperforming traditional layer-wise pruning strategies.

What Problem Does This Paper Address?

Deep learning models like VGG and AlexNet are notoriously "over-parameterized," containing millions or billions of parameters. While this redundancy helps during training, it makes deploying these models on resource-constrained devices (like smartphones or IoT sensors) nearly impossible due to high memory and computation costs.

Existing pruning methods often fall into two traps:

  1. Connection-level pruning: This creates "sparse" matrices. While mathematically efficient, standard hardware isn't optimized for sparsity, often leading to no real-world speedup.
  2. Layer-wise neuron pruning: This removes entire filters (making the network "thinner"), but it requires researchers to manually decide exactly how many neurons to cut from each specific layer—a tedious and often sub-optimal "guess-and-check" process.

The authors aim to automate this by asking: Can we just rank every neuron in the entire network and cut the least useful ones globally?

Methods

The authors propose a "select-prune-fine-tune" loop that operates on the whole network at once.

1. Global Scoring and Bias Correction

To compare a neuron in Layer 1 with a neuron in Layer 10, you need a fair metric. The authors tested three:

  • Mean Response ($\bar{R}$): Average output of a neuron.
  • Standard Deviation of Response ($\sigma(R)$): How much a neuron's output changes across different inputs.
  • Average Absolute Weight Sum (AAWS): The average magnitude of the weights connected to a neuron.

They discovered a positional bias: neurons in early layers often have much higher raw scores than those in later layers. To fix this, they introduced a Modified Contribution Score: they divide each neuron's score by the average score of its parent layer. This "flattens" the distribution, allowing for a fair global ranking.

2. The "Gradual" Approach

Instead of cutting 50% of the neurons in one go (which would destroy the network's performance), they use a pruning ratio ($r$)—typically 5%. In each step, they:

  1. Rank all neurons globally.
  2. Remove the bottom 5%.
  3. Perform a quick "fine-tuning" (re-training) to help the remaining neurons compensate for the loss.
  4. Repeat until the performance drops below a target threshold.

Datasets

The researchers evaluated their method using a VGG-like network (similar to VGG-16 but with extra BatchNormalization layers) trained on the CIFAR-10 image classification dataset.

Key Insights and Ideas

  • AAWS is the Winner: Among the metrics tested, the Average Absolute Weight Sum (AAWS) performed best. Crucially, AAWS is "data-independent," meaning you don't need to run a massive validation set through the model to calculate scores; you just look at the weights themselves.
  • Global > Local: Global pruning naturally discovers which layers are most redundant. In their experiments, the algorithm chose to prune higher-level layers more aggressively than low-level feature extractors, a decision that is hard to optimize manually in layer-wise pruning.
  • Stay Near the Optimum: By pruning only a small percentage ($r=0.05$) at a time, the model stays close to its "local optimal point." This makes the fine-tuning phase much faster and more stable than aggressive one-shot pruning.

Key Results

  • Efficiency: The GGP method successfully thinned the VGG-like network from 4,736 total neurons down to 3,310 while maintaining an accuracy of 86.76% (only a minor drop from the original 87.32%).
  • Superiority over Layer-wise: When compared to "proportional" layer-wise pruning (where every layer is thinned by the same percentage), the global approach consistently maintained higher accuracy at every stage of the pruning process.
  • Automatic Architecture Search: The method effectively "finds" a thinner sub-network architecture that is optimized for the specific task without human intervention.

Limitations and Future Work

  • Scope of Evaluation: The paper focuses primarily on CIFAR-10 and a VGG-style architecture. The authors note that testing on larger datasets like ImageNet or different architectures like ResNets would be a logical next step.
  • Metric Refinement: While AAWS worked well, the authors acknowledge that the modified contribution score is a heuristic. They suggest that future research could uncover even more sophisticated ways to evaluate a neuron's "global" importance.
  • Computational Overhead: While the final model is fast, the process of pruning (many rounds of fine-tuning) can be time-consuming during the development phase.

Summary updated

Abstract

Deep network pruning is an effective method to reduce the storage and computation cost of deep neural networks when applying them to resource-limited devices. Among many pruning granularities, neuron level pruning will remove redundant neurons and filters in the model and result in thinner networks. In this paper, we propose a gradually global pruning scheme for neuron level pruning. In each pruning step, a small percent of neurons were selected and dropped across all layers in the model. We also propose a simple method to eliminate the biases in evaluating the importance of neurons to make the scheme feasible. Compared with layer-wise pruning scheme, our scheme avoid the difficulty in determining the redundancy in each layer and is more effective for deep networks. Our scheme would automatically find a thinner sub-network in original network under a given performance.
BibTeX
@inproceedings{wang2017towards,
  author = {Zhengtao Wang and Ce Zhu and Zhiqiang Xia and Qi Guo and Yipeng Liu},
  title = {{Towards thinner convolutional neural networks through gradually global pruning}},
  booktitle = {ICIP},
  year = {2017},
  url = {https://openreview.net/forum?id=EOL5BAzrKb},
}

Loading related content...

Figure