Introduction
Magnitude-based pruning is a highly effective technique for compressing neural networks, but its success hinges on determining the correct "layerwise sparsity"—deciding exactly how many weights to cut from each individual layer.
Historically, researchers have relied on two extremes to allocate this sparsity budget. The first approach uses handcrafted heuristics, such as "leave the first layer dense," which often fail to generalize to novel architectures. The second approach relies on extensive search algorithms, like Reinforcement Learning or massive hyperparameter sweeps, to find the best ratios. While effective, these searches are computationally exorbitant.
To bridge this gap, this paper introduces LAMP (Layer-Adaptive Magnitude-based Pruning), a simple, hyperparameter-free importance score that automatically determines the optimal sparsity for every layer. LAMP aims to be as simple to implement as standard magnitude pruning while matching or exceeding the performance of expensive, optimized search schemes.
The LAMP Method
The core contribution of the paper is the LAMP score, a novel importance metric designed for global pruning. Instead of evaluating a weight solely by its absolute value, the LAMP score rescales the squared magnitude of a weight by the sum of all "surviving" squared weights in that same layer that are larger than or equal to it.
Mathematically, for a sorted weight vector $W$, the score is defined as: $$score(u; W) := \frac{(W[u])^2}{\sum_{v \ge u} (W[v])^2}$$
Why does this work? * Distortion Minimization: The score is theoretically grounded in minimizing the $\ell_2$ distortion (the error) introduced to the model's output when a specific weight is removed. * Global Thresholding: By calculating this score for every weight across the entire network, practitioners can set a single global threshold. Weights with scores below that threshold are pruned. Because the scores are normalized per layer, this single threshold automatically yields adaptive, heterogeneous sparsity levels across different layers. * Efficiency: LAMP requires no training, no hyperparameters, and adds negligible computational overhead compared to standard magnitude pruning.
Key Insights and Discoveries
The researchers evaluated LAMP across a wide range of computer vision (CIFAR, SVHN, ImageNet) and natural language processing tasks (Penn Treebank, WikiText-2), utilizing architectures ranging from ResNets and EfficientNets to Transformer-XL.
Through these experiments, several fascinating properties of LAMP emerged:
- Automatic Heuristic Recovery: LAMP naturally "discovers" common human-designed heuristics without being programmed to do so. For example, it inherently tends to keep the first and last layers of a network denser, validating common manual practices in the pruning community.
- Uniformity at the Limit: At extreme sparsity levels, LAMP tends to keep the number of surviving weights relatively uniform across layers, rather than keeping the ratio of surviving weights uniform. The authors conjecture this behavior is essential for maintaining the "memory capacity" of a highly sparse network.
- Robustness to Weight Rewinding: Unlike some baselines (such as the Erdös-Rényi kernel), LAMP remains highly effective even when using "weight rewinding"—resetting weights to their early-training values—a popular technique in Lottery Ticket Hypothesis research.
- Smart Filter Pruning: On architectures like ResNet-50, LAMP automatically prunes $3 \times 3$ convolutions more aggressively than $1 \times 1$ convolutions, recognizing that larger filters often harbor more redundancy.
Empirical Results
Across almost all vision benchmarks, LAMP provided a strictly superior sparsity-vs-accuracy tradeoff compared to Global Magnitude Pruning, Uniform Pruning, and the Erdös-Rényi kernel.
The performance gap was most pronounced on modern, complex architectures like EfficientNet-B0. For instance, on CIFAR-10, LAMP maintained an impressive 88.1% accuracy with only 1.44% of its weights remaining, whereas the strongest baseline plummeted to 77.8% under the same constraints. Furthermore, even in "one-shot" pruning scenarios (pruning once without iterative retraining), LAMP significantly outperformed competing methods, proving its viability for rapid model compression.
Limitations and Future Directions
While highly effective, the authors note a few limitations. First, LAMP focuses on unstructured pruning (removing individual weights). While this achieves high theoretical compression, it does not always translate to immediate inference speedups on standard hardware (like GPUs) without specialized sparse kernels.
Second, while LAMP was the top performer for Transformers on language tasks, the performance gains over baselines were marginal. The authors suspect this is because Transformer layers are already quite uniform in width compared to convolutional networks, leaving less room for layer-adaptive optimization. Finally, while the method is motivated by minimizing $\ell_2$ distortion, the exact theoretical link between minimizing output distortion and final retrained accuracy remains an area for future exploration.
Summary updated