TL;DR
This paper challenges the common belief that retraining pruned neural networks is a "computationally impractical" necessity. By reframing retraining as a Budgeted Training problem, the authors show that a simple linear learning rate schedule can drastically shorten the retraining phase—often by 10x or more—while outperforming complex state-of-the-art methods that try to avoid "hard" pruning altogether.
What Problem Does This Paper Address?
In the world of deep learning, "pruning" is the art of removing redundant weights to make models smaller and faster. The classic approach, Iterative Magnitude Pruning (IMP), involves a cycle: train, prune the smallest weights, and retrain to recover lost accuracy.
However, IMP has a bad reputation for being slow. Because it often requires hundreds of retraining epochs to reach peak performance, researchers have pivoted toward "pruning-stable" methods—complex algorithms that try to bake sparsity into the initial training process to avoid a separate retraining phase. The authors of this paper ask: Is retraining actually the problem, or have we just been using the wrong learning rate schedules?
Methods
The authors bridge the gap between pruning and Budgeted Training (the study of optimizing models within a strictly limited number of iterations). Their core contribution is a suite of efficient retraining strategies:
- Linear Learning Rate (LLR): Instead of complex stepped or cosine schedules, they use a simple linear decay to zero. In budgeted scenarios, linear schedules often outperform others.
- Adaptive Linear Learning Rate Restarting (ALLR): This is their "smart" version. It adaptively selects the starting learning rate for the retraining phase based on two factors:
- Weight Distortion ($d_1$): How much the weights changed during the pruning step (measured via $L_2$ norm).
- Time Budget ($d_2$): How much retraining time is available relative to the original training time.
- Budgeted IMP (BIMP): They propose a full end-to-end pipeline that applies these budgeted principles to the entire lifecycle of the model, including the initial dense training.
Datasets
The authors conducted an extensive evaluation across diverse tasks and architectures to ensure their findings weren't specific to one niche:
- Image Classification: ImageNet (ResNet-50, MaxViT) and CIFAR-10/100 (ResNet-18, ResNet-56, VGG-16, WideResNet).
- Semantic Segmentation: COCO (DeepLabV3) and CityScapes (PSPNet).
- Natural Language Translation: WMT16 EN-DE (T5-small).
Key Insights and Ideas
- Retraining is just "Short Training": The "aha moment" is realizing that the retraining phase is simply training under a tight budget. Previous heuristics (like "Learning Rate Rewinding") were accidentally approximating good budgeted training practices.
- The Linear Advantage: While cosine or stepped schedules are standard for long training runs, linear decay is mathematically and empirically superior when you only have a few epochs to work with.
- "Hard" Pruning is Fine: The community has spent years trying to avoid the "shock" of suddenly deleting weights. This paper suggests that the shock isn't the issue; the issue was failing to use a high enough learning rate immediately after the shock to help the model navigate to a new local minimum.
- Simplicity Wins: BIMP is essentially "IMP with a better clock." It is easier to implement than complex penalty-based or dynamic sparsity methods and often yields better accuracy-to-FLOPs tradeoffs.
Key Results
- Massive Speedups: On ImageNet with a ResNet-50, the authors showed that retraining can be shortened to just 2–10 epochs (down from the standard 90) with negligible loss in accuracy when using ALLR.
- Outperforming the SOTA: BIMP consistently outperformed or matched high-profile "pruning-stable" methods like GMP, DPF, and DNW. For example, on CIFAR-10 at 95% sparsity, BIMP achieved 92.57% accuracy, surpassing almost all competitors while being significantly faster to train (higher images-per-second throughput).
- Robustness: The ALLR method proved robust across different tasks. Whether it was a Transformer for translation or a CNN for segmentation, the adaptive linear schedule consistently found high-performing sparse states.
Limitations and Future Work
- Magnitude Pruning Focus: The study primarily focuses on magnitude-based pruning. While this is the most common form, it remains to be seen if these budgeted insights apply equally to more exotic pruning criteria (like gradient-based or Hessian-based pruning).
- Hyperparameter Sensitivity: While ALLR reduces the need for tuning, the initial "dense" budget ($T_0$) in the BIMP framework is still a hyperparameter that requires selection.
- Hardware Realities: The paper focuses on "theoretical speedup" (FLOP reduction). The authors acknowledge that translating these theoretical gains into actual wall-clock time on specific hardware (like GPUs) depends on the underlying sparse kernels available.
In summary, this paper is a refreshing reminder that sometimes the "old" way isn't broken—it just needs a better tuning strategy. By embracing retraining rather than fleeing from it, we can achieve state-of-the-art sparsity with a fraction of the computational headache.
Summary updated