Abstract
Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity
Introduction
Neural network pruning—the removal of redundant weights to optimize models for deployment—traditionally relies on weight magnitude as a proxy for importance. However, a significant challenge remains in determining the optimal layerwise sparsity. While global magnitude pruning is a common baseline, it often fails to account for the varying sensitivity of different layers, potentially leading to significant performance degradation. Conversely, uniform pruning ignores the inherent redundancy differences across layers. To address this, practitioners often employ handcrafted heuristics or computationally expensive searches. This paper introduces Layer-Adaptive Magnitude-based Pruning (LAMP), a theoretically grounded, automated approach to layerwise sparsity selection.
Methodology: The LAMP Score
The LAMP score redefines weight importance by considering a weight's magnitude relative to all larger weights within the same layer. This normalization is derived from the goal of minimizing the $\ell_2$ distortion of the model's output.
For a layer with weights $W$ sorted by magnitude, the LAMP score for the $u$-th weight is: $$score(u; W) := \frac{(W[u])^2}{\sum_{v \ge u} (W[v])^2}$$
By computing these scores across all layers and applying a global threshold, LAMP automatically determines the appropriate sparsity for each layer. This process naturally recovers common human-designed heuristics, such as maintaining higher density in the first and last layers, without requiring manual tuning.
Experimental Results
The effectiveness of LAMP was validated across a broad spectrum of tasks, including image classification (CIFAR, SVHN, Restricted ImageNet) and language modeling (Penn Treebank, WikiText-2). Architectures tested ranged from standard ResNets and VGGs to more complex models like EfficientNet-B0 and Transformer-XL.
As shown in the results, LAMP consistently achieves a superior tradeoff between sparsity and accuracy. On EfficientNet-B0, LAMP maintained an impressive 88.1% accuracy with only 1.44% of weights remaining, significantly outperforming the Erdos-Renyi baseline which dropped to 77.8% at the same level. LAMP also proved robust in weight-rewinding scenarios, a key component of the Lottery Ticket Hypothesis, where other adaptive methods often struggle.
Conclusion
LAMP offers a simple, efficient, and hyperparameter-free method for optimizing layerwise sparsity in neural network pruning. By incorporating local context into a global pruning framework, it outperforms both simple baselines and complex handcrafted heuristics. While its impact on Transformers is more subtle due to their architectural uniformity, its performance on diverse CNN architectures demonstrates its value as a general-purpose pruning tool. Future research will investigate the theoretical properties of the uniform weight distribution observed at extreme sparsities and explore extensions to structured pruning.
Summary updated