TL;DR
This paper introduces "Annealed Pruning" (AP), a novel weight pruning technique for Artificial Neural Networks (ANNs) that not only drastically reduces model size (by up to 85%) but also improves generalization performance, outperforming the popular dropout regularization method. The key insight is that pruning large weights, rather than small ones, is crucial for this generalization boost.
What Problem Does This Paper Address?
Modern Artificial Neural Networks (ANNs) achieve remarkable performance, especially in computer vision, but their sheer size (millions of parameters) makes them impractical for deployment on resource-constrained devices like mobile phones. This leads to high computational and memory costs. While parameter pruning offers a way to shrink ANNs, existing methods primarily focus on reducing model size without significantly improving their ability to generalize to unseen data (i.e., avoiding overfitting). This paper aims to bridge this gap by developing a pruning technique that simultaneously reduces model size and enhances generalization.
Methods
The core of the paper is the Annealed Pruning (AP) algorithm. Unlike traditional pruning that targets small-magnitude weights, AP focuses on pruning large-magnitude weights. The algorithm works iteratively during the ANN training process. At the end of each training epoch, AP creates a binary mask that identifies weights to be pruned. Crucially, AP allows pruned weights a chance to "reenter" the network in subsequent iterations, inspired by genetic algorithms. This "reentry" mechanism is randomized and gradually reduced as training progresses, with weights permanently pruned only in the final stages. This annealing process, combined with targeting large weights, is hypothesized to strengthen the robustness of learned features, leading to better generalization.
Datasets
The primary dataset used for evaluation is CIFAR-10. This dataset consists of 60,000 32x32 pixel images across 10 classes, with 50,000 images for training and 10,000 for testing. The authors used this standard benchmark to compare their AP method against a baseline CNN and a CNN trained with dropout.
Key Insights and Ideas
- Pruning Large Weights for Generalization: The most significant finding is that targeting large-magnitude weights for pruning, rather than small ones, leads to improved generalization. This is counter-intuitive to many pruning approaches that remove "unimportant" small weights.
- Pruning as Regularization: The paper posits that pruning large weights acts as a form of regularization, similar to dropout, by forcing features to be more generally useful and robust.
- Annealed Reentry: The ability for pruned weights to be re-introduced into the network during training (with decreasing probability over time) is a novel aspect that likely contributes to the improved generalization. This prevents the premature removal of potentially important connections.
Key Results
- Significant Size Reduction: AP achieved an 85% reduction in the parameter count of the first dense layer of a CNN on the CIFAR-10 dataset.
- Improved Accuracy: The AP-trained CNN achieved a test-data accuracy of 78%, which is 5% higher than the baseline CNN (74.2%) and 1.5% higher than the dropout-regularized CNN (76.5%).
- Outperforming Dropout: AP demonstrated superior performance compared to dropout, a widely used regularization technique, in terms of both parameter reduction and generalization accuracy.
Limitations and Future Work
- Hyperparameter Sensitivity: The authors acknowledge that AP has several hyperparameters and that their current results could potentially be improved with different settings. They plan to investigate the sensitivity of their results to these hyperparameters and develop heuristics for choosing them.
- Interaction with Other Regularizers: The paper raises the question of whether AP can be effectively combined with other regularization techniques, such as dropout, to achieve even better results.
- Permanent Pruning Schedule: The authors are interested in exploring a modified AP that permanently removes weights as soon as they are selected for pruning, rather than allowing reentry. This could potentially lead to faster ANN training if sparse matrix operations are utilized.
Summary updated