How I Learned to Stop Worrying and Love Retraining

Max Zimmer, Christoph Spiegel, Sebastian PokuttaICLR 2023 · OpenReview

TL;DR

This paper challenges the common belief that retraining pruned neural networks is a "computationally impractical" necessity. By reframing retraining as a Budgeted Training problem, the authors show that a simple linear learning rate schedule can drastically shorten the retraining phase—often by 10x or more—while outperforming complex state-of-the-art methods that try to avoid "hard" pruning altogether.

What Problem Does This Paper Address?

In the world of deep learning, "pruning" is the art of removing redundant weights to make models smaller and faster. The classic approach, Iterative Magnitude Pruning (IMP), involves a cycle: train, prune the smallest weights, and retrain to recover lost accuracy.

However, IMP has a bad reputation for being slow. Because it often requires hundreds of retraining epochs to reach peak performance, researchers have pivoted toward "pruning-stable" methods—complex algorithms that try to bake sparsity into the initial training process to avoid a separate retraining phase. The authors of this paper ask: Is retraining actually the problem, or have we just been using the wrong learning rate schedules?

Methods

The authors bridge the gap between pruning and Budgeted Training (the study of optimizing models within a strictly limited number of iterations). Their core contribution is a suite of efficient retraining strategies:

  1. Linear Learning Rate (LLR): Instead of complex stepped or cosine schedules, they use a simple linear decay to zero. In budgeted scenarios, linear schedules often outperform others.
Comparison of learning rate schedules for retraining, highlighting the proposed Linear Learning Rate (LLR).
  1. Adaptive Linear Learning Rate Restarting (ALLR): This is their "smart" version. It adaptively selects the starting learning rate for the retraining phase based on two factors:
  2. Weight Distortion ($d_1$): How much the weights changed during the pruning step (measured via $L_2$ norm).
  3. Time Budget ($d_2$): How much retraining time is available relative to the original training time.
  4. Budgeted IMP (BIMP): They propose a full end-to-end pipeline that applies these budgeted principles to the entire lifecycle of the model, including the initial dense training.

Datasets

The authors conducted an extensive evaluation across diverse tasks and architectures to ensure their findings weren't specific to one niche:

  • Image Classification: ImageNet (ResNet-50, MaxViT) and CIFAR-10/100 (ResNet-18, ResNet-56, VGG-16, WideResNet).
  • Semantic Segmentation: COCO (DeepLabV3) and CityScapes (PSPNet).
  • Natural Language Translation: WMT16 EN-DE (T5-small).

Key Insights and Ideas

  • Retraining is just "Short Training": The "aha moment" is realizing that the retraining phase is simply training under a tight budget. Previous heuristics (like "Learning Rate Rewinding") were accidentally approximating good budgeted training practices.
  • The Linear Advantage: While cosine or stepped schedules are standard for long training runs, linear decay is mathematically and empirically superior when you only have a few epochs to work with.
  • "Hard" Pruning is Fine: The community has spent years trying to avoid the "shock" of suddenly deleting weights. This paper suggests that the shock isn't the issue; the issue was failing to use a high enough learning rate immediately after the shock to help the model navigate to a new local minimum.
  • Simplicity Wins: BIMP is essentially "IMP with a better clock." It is easier to implement than complex penalty-based or dynamic sparsity methods and often yields better accuracy-to-FLOPs tradeoffs.

Key Results

  • Massive Speedups: On ImageNet with a ResNet-50, the authors showed that retraining can be shortened to just 2–10 epochs (down from the standard 90) with negligible loss in accuracy when using ALLR.
Test accuracy deviation across different retraining budgets for ResNet-50 at 70%, 80%, and 90% sparsity. ALLR and LLR consistently outperform other schedules at low epoch counts.
  • Outperforming the SOTA: BIMP consistently outperformed or matched high-profile "pruning-stable" methods like GMP, DPF, and DNW. For example, on CIFAR-10 at 95% sparsity, BIMP achieved 92.57% accuracy, surpassing almost all competitors while being significantly faster to train (higher images-per-second throughput).
  • Robustness: The ALLR method proved robust across different tasks. Whether it was a Transformer for translation or a CNN for segmentation, the adaptive linear schedule consistently found high-performing sparse states.

Limitations and Future Work

  • Magnitude Pruning Focus: The study primarily focuses on magnitude-based pruning. While this is the most common form, it remains to be seen if these budgeted insights apply equally to more exotic pruning criteria (like gradient-based or Hessian-based pruning).
  • Hyperparameter Sensitivity: While ALLR reduces the need for tuning, the initial "dense" budget ($T_0$) in the BIMP framework is still a hyperparameter that requires selection.
  • Hardware Realities: The paper focuses on "theoretical speedup" (FLOP reduction). The authors acknowledge that translating these theoretical gains into actual wall-clock time on specific hardware (like GPUs) depends on the underlying sparse kernels available.

In summary, this paper is a refreshing reminder that sometimes the "old" way isn't broken—it just needs a better tuning strategy. By embracing retraining rather than fleeing from it, we can achieve state-of-the-art sparsity with a fraction of the computational headache.

Summary updated

Abstract

Many Neural Network Pruning approaches consist of several iterative training and pruning steps, seemingly losing a significant amount of their performance after pruning and then recovering it in the subsequent retraining phase. Recent works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance of the learning rate schedule during the retraining phase and propose specific heuristics for choosing such a schedule for IMP (Han et al., 2015). We place these findings in the context of the results of Li et al. (2020) regarding the training of models within a fixed training budget and demonstrate that, consequently, the retraining phase can be massively shortened using a simple linear learning rate schedule. Improving on existing retraining approaches, we additionally propose a method to adaptively select the initial value of the linear schedule. Going a step further, we propose similarly imposing a budget on the initial dense training phase and show that the resulting simple and efficient method is capable of outperforming significantly more complex or heavily parameterized state-of-the-art approaches that attempt to sparsify the network during training. These findings not only advance our understanding of the retraining phase, but more broadly question the belief that one should aim to avoid the need for retraining and reduce the negative effects of ‘hard’ pruning by incorporating the sparsification process into the standard training.
BibTeX
@inproceedings{zimmer2023how,
  author = {Max Zimmer and Christoph Spiegel and Sebastian Pokutta},
  title = {{How I Learned to Stop Worrying and Love Retraining}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=_nF5imFKQI},
}
Figure