Pruning Neural Networks at Initialization: Why Are We Missing the Mark?

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, Michael Carbin

Neural network pruning is a powerful technique for reducing the size and computational cost of AI models. Traditionally, pruning occurs after a model is fully trained. However, as models grow massive, the cost of the initial training phase itself has become a significant bottleneck. To address this, researchers have proposed "pruning at initialization"—identifying a sparse subnetwork before training even begins to save massive amounts of compute.

Several high-profile methods, such as SNIP, GraSP, and SynFlow, claim to find these "important" connections using gradients or Hessian information at the very start of training. But do these methods actually work as advertised, and are they truly finding the right individual weights? This paper critically evaluates these techniques and uncovers a surprising property about how they actually function.

Evaluating At-Initialization Pruning

The authors conduct a rigorous benchmarking of four main "at-initialization" pruning techniques across standard computer vision benchmarks (MNIST, CIFAR-10, TinyImageNet, and ImageNet):

  1. SNIP: Prunes weights that are least salient for the loss.
  2. GraSP: Prunes weights that most hinder "gradient flow."
  3. SynFlow: A data-independent iterative method designed to avoid "layer collapse" (where an entire layer is accidentally pruned to zero).
  4. Magnitude Pruning (at init): A simple baseline that prunes the smallest weights right at the start.
Accuracy of pruning methods across different sparsities

While SNIP, GraSP, and SynFlow all successfully beat a purely random pruning baseline, they consistently underperform compared to magnitude pruning conducted after training. There remains a significant performance gap, suggesting we are still missing the mark for finding truly optimal sparse networks at initialization.

The Shuffling Paradox

To test if these methods were truly identifying specific, critical connections, the authors introduced two clever ablations:

  • Random Shuffling: Taking the pruning mask generated by a method and randomly shuffling it within each layer. If the method was truly finding specific "important" connections, shuffling should destroy performance.
  • Reinitialization: Keeping the pruning mask but replacing the weight values with new random ones.

The most striking finding of the paper is the Shuffling Paradox: randomly shuffling the pruning masks within each layer does not hurt the accuracy of these at-initialization methods. In fact, for SynFlow, shuffling actually improves performance at high sparsities.

Ablation results showing the effect of random shuffling and reinitialization

This visualizes a critical flaw: when pruning after training (bottom row), shuffling the mask destroys accuracy. But for at-initialization methods (top rows), the shuffled masks perform just as well as the original ones.

This implies that the "intelligence" of SNIP, GraSP, and SynFlow isn't in picking specific individual weights. Instead, their success relies almost entirely on deciding the fraction of weights to prune in each layer (the layer-wise sparsity ratio). The per-weight pruning decisions made by these state-of-the-art methods can be replaced by a simple per-layer choice without any loss in accuracy.

Additional Insights

  • Magnitude Pruning is a Strong Baseline: Simple magnitude pruning at initialization—often ignored in previous papers—is surprisingly competitive with much more complex, gradient-based heuristics.
  • Neuron Collapse: The authors identify a pathology in SynFlow called "neuron collapse," where the method prunes entire convolutional channels. Shuffling helps mitigate this because it redistributes those pruned weights, preventing the "death" of entire neurons.
  • Data Independence: SynFlow and Magnitude pruning are competitive even without looking at a single piece of training data. This further suggests that the specific data-driven "importance" signals used by SNIP and GraSP aren't as vital as previously thought.

Where Do We Go From Here?

This paper provides a crucial reality check for the field of sparse training. While it identifies why current methods are failing to reach their full potential, it doesn't provide a silver bullet replacement. It proves that current heuristics are essentially just sophisticated ways to pick layer-wise sparsity ratios.

Future work must focus on finding new pruning signals that are genuinely sensitive to specific connections rather than just layer-wise statistics. Furthermore, since pruning at the very first iteration (step 0) seems incredibly difficult, exploring "early-training" pruning (after a few hundred steps) might be a more viable middle ground. Bridging the gap between at-initialization heuristics and retroactive "Lottery Ticket" masks remains the holy grail of sparse neural network training.

Summary updated

Abstract

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.
BibTeX
@inproceedings{frankle2021pruning,
  author = {Jonathan Frankle and Gintare Karolina Dziugaite and Daniel Roy and Michael Carbin},
  title = {{Pruning Neural Networks at Initialization: Why Are We Missing the Mark?}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=OGf9MGmWmhz},
}
Figure