Layer-adaptive Sparsity for the Magnitude-based Pruning

Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, Jinwoo Shin

Introduction

Magnitude-based pruning is a highly effective technique for compressing neural networks, but its success hinges on determining the correct "layerwise sparsity"—deciding exactly how many weights to cut from each individual layer.

Historically, researchers have relied on two extremes to allocate this sparsity budget. The first approach uses handcrafted heuristics, such as "leave the first layer dense," which often fail to generalize to novel architectures. The second approach relies on extensive search algorithms, like Reinforcement Learning or massive hyperparameter sweeps, to find the best ratios. While effective, these searches are computationally exorbitant.

To bridge this gap, this paper introduces LAMP (Layer-Adaptive Magnitude-based Pruning), a simple, hyperparameter-free importance score that automatically determines the optimal sparsity for every layer. LAMP aims to be as simple to implement as standard magnitude pruning while matching or exceeding the performance of expensive, optimized search schemes.

The LAMP Method

The core contribution of the paper is the LAMP score, a novel importance metric designed for global pruning. Instead of evaluating a weight solely by its absolute value, the LAMP score rescales the squared magnitude of a weight by the sum of all "surviving" squared weights in that same layer that are larger than or equal to it.

Mathematically, for a sorted weight vector $W$, the score is defined as: $$score(u; W) := \frac{(W[u])^2}{\sum_{v \ge u} (W[v])^2}$$

The LAMP pruning pipeline: sorted weight magnitudes are converted to LAMP scores before global pooling and pruning.

Why does this work? * Distortion Minimization: The score is theoretically grounded in minimizing the $\ell_2$ distortion (the error) introduced to the model's output when a specific weight is removed. * Global Thresholding: By calculating this score for every weight across the entire network, practitioners can set a single global threshold. Weights with scores below that threshold are pruned. Because the scores are normalized per layer, this single threshold automatically yields adaptive, heterogeneous sparsity levels across different layers. * Efficiency: LAMP requires no training, no hyperparameters, and adds negligible computational overhead compared to standard magnitude pruning.

Key Insights and Discoveries

The researchers evaluated LAMP across a wide range of computer vision (CIFAR, SVHN, ImageNet) and natural language processing tasks (Penn Treebank, WikiText-2), utilizing architectures ranging from ResNets and EfficientNets to Transformer-XL.

Through these experiments, several fascinating properties of LAMP emerged:

  • Automatic Heuristic Recovery: LAMP naturally "discovers" common human-designed heuristics without being programmed to do so. For example, it inherently tends to keep the first and last layers of a network denser, validating common manual practices in the pruning community.
  • Uniformity at the Limit: At extreme sparsity levels, LAMP tends to keep the number of surviving weights relatively uniform across layers, rather than keeping the ratio of surviving weights uniform. The authors conjecture this behavior is essential for maintaining the "memory capacity" of a highly sparse network.
  • Robustness to Weight Rewinding: Unlike some baselines (such as the Erdös-Rényi kernel), LAMP remains highly effective even when using "weight rewinding"—resetting weights to their early-training values—a popular technique in Lottery Ticket Hypothesis research.
  • Smart Filter Pruning: On architectures like ResNet-50, LAMP automatically prunes $3 \times 3$ convolutions more aggressively than $1 \times 1$ convolutions, recognizing that larger filters often harbor more redundancy.

Empirical Results

Across almost all vision benchmarks, LAMP provided a strictly superior sparsity-vs-accuracy tradeoff compared to Global Magnitude Pruning, Uniform Pruning, and the Erdös-Rényi kernel.

LAMP consistently outperforms baseline pruning methods in test accuracy at high sparsity levels.

The performance gap was most pronounced on modern, complex architectures like EfficientNet-B0. For instance, on CIFAR-10, LAMP maintained an impressive 88.1% accuracy with only 1.44% of its weights remaining, whereas the strongest baseline plummeted to 77.8% under the same constraints. Furthermore, even in "one-shot" pruning scenarios (pruning once without iterative retraining), LAMP significantly outperformed competing methods, proving its viability for rapid model compression.

Limitations and Future Directions

While highly effective, the authors note a few limitations. First, LAMP focuses on unstructured pruning (removing individual weights). While this achieves high theoretical compression, it does not always translate to immediate inference speedups on standard hardware (like GPUs) without specialized sparse kernels.

Second, while LAMP was the top performer for Transformers on language tasks, the performance gains over baselines were marginal. The authors suspect this is because Transformer layers are already quite uniform in width compared to convolutional networks, leaving less room for layer-adaptive optimization. Finally, while the method is motivated by minimizing $\ell_2$ distortion, the exact theoretical link between minimizing output distortion and final retrained accuracy remains an area for future exploration.

Summary updated

Abstract

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on ``how to choose,'' the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity
BibTeX
@inproceedings{lee2021layer,
  author = {Jaeho Lee and Sejun Park and Sangwoo Mo and Sungsoo Ahn and Jinwoo Shin},
  title = {{Layer-adaptive Sparsity for the Magnitude-based Pruning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=Ro67KoHp5Y},
}
Figure