PaI is getting competitive by training longer

Advait Gadhikar, Sree Harsha Nelaturu, Rebekka BurkholzOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

The success of iterative pruning methods in achieving state-of-the-art sparse networks has largely been attributed to improved mask identification and an implicit regularization induced by pruning. We challenge this hypothesis and instead posit that their increased training epochs enable improved optimization. To verify this, we show that pruning at initialization (PaI) is significantly boosted by increased training epochs with repeating (cyclic) learning rate schedules akin to iterative pruning, even outperforming standard iterative pruning methods. The dominant mechanism how this is achieved, as we conjecture, can be attributed to a better exploration of the loss landscape leading to a lower training loss. However, at high sparsity, increased training alone is not enough for competitive performance. A strong coupling between learnt parameter initialization and mask seems to be required. Standard methods obtain this coupling via expensive pruning-training iterations, starting from a dense network. To achieve this with sparse training instead, we propose SCULPT-ing, i.e., cyclic training of any sparse mask followed by a single pruning step to couple the parameters and the mask, which is able to match the performance of state-of-the-art iterative pruning methods in the high sparsity regime at reduced computational cost.
BibTeX
@misc{gadhikar2025pai,
  author = {Advait Gadhikar and Sree Harsha Nelaturu and Rebekka Burkholz},
  title = {{PaI is getting competitive by training longer}},
  year = {2025},
  url = {https://openreview.net/forum?id=8s1GMWsLlj},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure