Pruning Neural Networks at Initialization: Why are We Missing the Mark?

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin

Abstract

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP, GraSP, SynFlow, and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. This suggests that current methods primarily discover effective per-layer pruning ratios rather than specific important connections.

The Quest for Early Pruning

Deep learning models are notoriously overparameterized, leading to massive computational costs. While "pruning"—the process of removing unnecessary weights—is highly effective after a model is trained, the holy grail is pruning at initialization. If we could identify and remove 90% of a network's weights before the first epoch, we could slash training time, energy consumption, and hardware requirements.

Several recent algorithms (SNIP, GraSP, and SynFlow) claim to do exactly this by using gradients or "synaptic strengths" to find a "winning ticket" subnetwork at the very start. This paper investigates whether these methods are actually finding specific, structurally important connections or if they are simply performing a sophisticated version of "layer-wise" random pruning.

Methodology: Stress-Testing the Heuristics

The authors conduct a rigorous "stress test" of four pruning-at-init methods: SNIP (saliency-based), GraSP (gradient flow-based), SynFlow (data-independent path-based), and simple Magnitude Pruning.

To see if these methods truly identify critical weights, the authors introduce two key "ablations":

  1. Random Shuffling: They take the pruning mask generated by a method and randomly shuffle it within each layer. This preserves the fraction of weights pruned in each layer but changes which specific weights are kept.
  2. Reinitialization: They keep the pruning mask but replace the weight values with new random samples from the original distribution.

If a method has truly identified a specific, important subnetwork structure, these changes should significantly degrade the model's final accuracy.

Key Insights: Layers, Not Weights

The most striking finding is that for all "at-init" methods, shuffling the weights within a layer preserves or even improves final accuracy. This means the "intelligence" of these algorithms is limited to deciding how much to prune each layer (e.g., prune 50% of Layer 1 and 90% of Layer 10), rather than identifying specific critical connections.

Figure 3: Shuffling and reinitializing the pruning mask has little effect on at-init methods (top four rows), but causes accuracy to collapse for post-training magnitude pruning (bottom row).
  • Initialization Independence: These subnetworks are insensitive to their initial weight values. You can throw away the weights used to calculate the mask, re-randomize the subnetwork, and it will still train to the same accuracy.
  • The "Post-Init" Shift: The paper shows that pruning only becomes sensitive to specific weight structures (where shuffling hurts) after the network has been trained for a significant number of iterations. This suggests that the information required to find truly "important" weights may not even exist at the moment of initialization.
  • SynFlow’s Pathology: The authors discovered that SynFlow often suffers from "neuron collapse," where it accidentally prunes every single connection to a specific neuron. Interestingly, shuffling the mask actually fixes this issue, explaining why shuffling sometimes improves SynFlow's performance.

Key Results: The Performance Gap

While SNIP, GraSP, and SynFlow beat a baseline of global random pruning, they consistently fail to match the accuracy of simple magnitude pruning performed after training.

Figure 2: Test accuracy across different architectures and datasets. Pruning at initialization (SNIP, GraSP, SynFlow, Magnitude) consistently lags behind magnitude pruning performed after training.

In every tested configuration at initialization, a subnetwork with a shuffled mask performed as well as the original. In contrast, shuffling a network pruned after training (using magnitude pruning) caused accuracy to plummet (e.g., from ~90% to ~10% on CIFAR-10). Furthermore, SynFlow and Magnitude Pruning at init are competitive even without looking at any training data, further supporting the idea that these methods are just finding coarse layer-wise proportions.

Limitations and Future Work

The primary limitation is that the paper is a "negative" result—it successfully debunks the current understanding of how these methods work but does not provide a new, superior method for pruning at initialization.

The authors suggest that future research should move away from looking at initialization alone. Instead, we should look for "new signals" that emerge early in training. The fact that Lottery Ticket Rewinding (LTR)—which prunes slightly after initialization—performs so much better suggests there is a "sweet spot" early in the training process where the network's structural importance finally becomes visible. Identifying exactly when and why that happens is the next big challenge for the field.

Summary updated

Abstract

Recent work has explored the possibility of pruning neural networks at initialization. We assess proposals for doing so: SNIP (Lee et al., 2019), GraSP (Wang et al., 2020), SynFlow (Tanaka et al., 2020), and magnitude pruning. Although these methods surpass the trivial baseline of random pruning, they remain below the accuracy of magnitude pruning after training, and we endeavor to understand why. We show that, unlike pruning after training, randomly shuffling the weights these methods prune within each layer or sampling new initial values preserves or improves accuracy. As such, the per-weight pruning decisions made by these methods can be replaced by a per-layer choice of the fraction of weights to prune. This property suggests broader challenges with the underlying pruning heuristics, the desire to prune at initialization, or both.
BibTeX
@inproceedings{frankle2021pruning,
  author = {Jonathan Frankle and Gintare Karolina Dziugaite and Daniel M. Roy and Michael Carbin},
  title = {{Pruning Neural Networks at Initialization: Why are We Missing the Mark?}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=OGf9MGmWmhz},
}
Figure