TL;DR
This paper introduces Gradually Global Pruning (GGP), a method that thins out Convolutional Neural Networks (CNNs) by removing redundant neurons across the entire model simultaneously rather than layer-by-layer. By using a simple normalization technique to compare neuron importance across different layers, the authors achieve significant model size reduction while maintaining high accuracy, outperforming traditional layer-wise pruning strategies.
What Problem Does This Paper Address?
Deep learning models like VGG and AlexNet are notoriously "over-parameterized," containing millions or billions of parameters. While this redundancy helps during training, it makes deploying these models on resource-constrained devices (like smartphones or IoT sensors) nearly impossible due to high memory and computation costs.
Existing pruning methods often fall into two traps:
- Connection-level pruning: This creates "sparse" matrices. While mathematically efficient, standard hardware isn't optimized for sparsity, often leading to no real-world speedup.
- Layer-wise neuron pruning: This removes entire filters (making the network "thinner"), but it requires researchers to manually decide exactly how many neurons to cut from each specific layer—a tedious and often sub-optimal "guess-and-check" process.
The authors aim to automate this by asking: Can we just rank every neuron in the entire network and cut the least useful ones globally?
Methods
The authors propose a "select-prune-fine-tune" loop that operates on the whole network at once.
1. Global Scoring and Bias Correction
To compare a neuron in Layer 1 with a neuron in Layer 10, you need a fair metric. The authors tested three:
- Mean Response ($\bar{R}$): Average output of a neuron.
- Standard Deviation of Response ($\sigma(R)$): How much a neuron's output changes across different inputs.
- Average Absolute Weight Sum (AAWS): The average magnitude of the weights connected to a neuron.
They discovered a positional bias: neurons in early layers often have much higher raw scores than those in later layers. To fix this, they introduced a Modified Contribution Score: they divide each neuron's score by the average score of its parent layer. This "flattens" the distribution, allowing for a fair global ranking.
2. The "Gradual" Approach
Instead of cutting 50% of the neurons in one go (which would destroy the network's performance), they use a pruning ratio ($r$)—typically 5%. In each step, they:
- Rank all neurons globally.
- Remove the bottom 5%.
- Perform a quick "fine-tuning" (re-training) to help the remaining neurons compensate for the loss.
- Repeat until the performance drops below a target threshold.
Datasets
The researchers evaluated their method using a VGG-like network (similar to VGG-16 but with extra BatchNormalization layers) trained on the CIFAR-10 image classification dataset.
Key Insights and Ideas
- AAWS is the Winner: Among the metrics tested, the Average Absolute Weight Sum (AAWS) performed best. Crucially, AAWS is "data-independent," meaning you don't need to run a massive validation set through the model to calculate scores; you just look at the weights themselves.
- Global > Local: Global pruning naturally discovers which layers are most redundant. In their experiments, the algorithm chose to prune higher-level layers more aggressively than low-level feature extractors, a decision that is hard to optimize manually in layer-wise pruning.
- Stay Near the Optimum: By pruning only a small percentage ($r=0.05$) at a time, the model stays close to its "local optimal point." This makes the fine-tuning phase much faster and more stable than aggressive one-shot pruning.
Key Results
- Efficiency: The GGP method successfully thinned the VGG-like network from 4,736 total neurons down to 3,310 while maintaining an accuracy of 86.76% (only a minor drop from the original 87.32%).
- Superiority over Layer-wise: When compared to "proportional" layer-wise pruning (where every layer is thinned by the same percentage), the global approach consistently maintained higher accuracy at every stage of the pruning process.
- Automatic Architecture Search: The method effectively "finds" a thinner sub-network architecture that is optimized for the specific task without human intervention.
Limitations and Future Work
- Scope of Evaluation: The paper focuses primarily on CIFAR-10 and a VGG-style architecture. The authors note that testing on larger datasets like ImageNet or different architectures like ResNets would be a logical next step.
- Metric Refinement: While AAWS worked well, the authors acknowledge that the modified contribution score is a heuristic. They suggest that future research could uncover even more sophisticated ways to evaluate a neuron's "global" importance.
- Computational Overhead: While the final model is fast, the process of pruning (many rounds of fine-tuning) can be time-consuming during the development phase.
Summary updated