Neural network pruning is a powerful technique for reducing the size and computational cost of AI models. Traditionally, pruning occurs after a model is fully trained. However, as models grow massive, the cost of the initial training phase itself has become a significant bottleneck. To address this, researchers have proposed "pruning at initialization"—identifying a sparse subnetwork before training even begins to save massive amounts of compute.
Several high-profile methods, such as SNIP, GraSP, and SynFlow, claim to find these "important" connections using gradients or Hessian information at the very start of training. But do these methods actually work as advertised, and are they truly finding the right individual weights? This paper critically evaluates these techniques and uncovers a surprising property about how they actually function.
Evaluating At-Initialization Pruning
The authors conduct a rigorous benchmarking of four main "at-initialization" pruning techniques across standard computer vision benchmarks (MNIST, CIFAR-10, TinyImageNet, and ImageNet):
- SNIP: Prunes weights that are least salient for the loss.
- GraSP: Prunes weights that most hinder "gradient flow."
- SynFlow: A data-independent iterative method designed to avoid "layer collapse" (where an entire layer is accidentally pruned to zero).
- Magnitude Pruning (at init): A simple baseline that prunes the smallest weights right at the start.
While SNIP, GraSP, and SynFlow all successfully beat a purely random pruning baseline, they consistently underperform compared to magnitude pruning conducted after training. There remains a significant performance gap, suggesting we are still missing the mark for finding truly optimal sparse networks at initialization.
The Shuffling Paradox
To test if these methods were truly identifying specific, critical connections, the authors introduced two clever ablations:
- Random Shuffling: Taking the pruning mask generated by a method and randomly shuffling it within each layer. If the method was truly finding specific "important" connections, shuffling should destroy performance.
- Reinitialization: Keeping the pruning mask but replacing the weight values with new random ones.
The most striking finding of the paper is the Shuffling Paradox: randomly shuffling the pruning masks within each layer does not hurt the accuracy of these at-initialization methods. In fact, for SynFlow, shuffling actually improves performance at high sparsities.
This visualizes a critical flaw: when pruning after training (bottom row), shuffling the mask destroys accuracy. But for at-initialization methods (top rows), the shuffled masks perform just as well as the original ones.
This implies that the "intelligence" of SNIP, GraSP, and SynFlow isn't in picking specific individual weights. Instead, their success relies almost entirely on deciding the fraction of weights to prune in each layer (the layer-wise sparsity ratio). The per-weight pruning decisions made by these state-of-the-art methods can be replaced by a simple per-layer choice without any loss in accuracy.
Additional Insights
- Magnitude Pruning is a Strong Baseline: Simple magnitude pruning at initialization—often ignored in previous papers—is surprisingly competitive with much more complex, gradient-based heuristics.
- Neuron Collapse: The authors identify a pathology in SynFlow called "neuron collapse," where the method prunes entire convolutional channels. Shuffling helps mitigate this because it redistributes those pruned weights, preventing the "death" of entire neurons.
- Data Independence: SynFlow and Magnitude pruning are competitive even without looking at a single piece of training data. This further suggests that the specific data-driven "importance" signals used by SNIP and GraSP aren't as vital as previously thought.
Where Do We Go From Here?
This paper provides a crucial reality check for the field of sparse training. While it identifies why current methods are failing to reach their full potential, it doesn't provide a silver bullet replacement. It proves that current heuristics are essentially just sophisticated ways to pick layer-wise sparsity ratios.
Future work must focus on finding new pruning signals that are genuinely sensitive to specific connections rather than just layer-wise statistics. Furthermore, since pruning at the very first iteration (step 0) seems incredibly difficult, exploring "early-training" pruning (after a few hundred steps) might be a more viable middle ground. Bridging the gap between at-initialization heuristics and retroactive "Lottery Ticket" masks remains the holy grail of sparse neural network training.
Summary updated