The Generalization-Stability Tradeoff In Neural Network Pruning

Brian R. Bartoldson, Ari S. Morcos, Adrian Barbu, Gordon Erlebacher

TL;DR

Neural network pruning is often prized for its ability to compress models, but recent research reveals its hidden power as a regularizer. A newly discovered "generalization-stability tradeoff" shows that the more a pruning event "shocks" a network—causing a temporary drop in accuracy—the better the model ultimately generalizes. This instability acts similarly to noise injection, forcing the network into flatter, more robust regions of the loss landscape.

The Paradox of Parameter Counts

In deep learning, we face a strange contradiction. Modern theory suggests that overparameterization (adding more weights) helps models generalize better. Conversely, pruning (removing weights) is a well-known technique that also improves generalization.

If both adding and removing parameters help, what is actually driving the performance boost? The authors set out to determine if pruning’s benefits stem from the final reduced parameter count, or from the process of pruning itself. Specifically, they investigate whether the "instability" introduced during pruning—the sudden dip in performance when weights vanish—acts as a form of regularization.

Measuring the "Shock" of Pruning

The researchers analyzed the behavior of Magnitude Pruning across various convolutional architectures (VGG11, ResNet18, ResNet20, and ResNet56) using CIFAR-10 and CIFAR-100 datasets.

To quantify the effect of pruning, they introduced two key metrics:

  • Pruning Instability: The percentage drop in test accuracy immediately following a pruning event.
  • Pruning Stability: The fraction of accuracy retained ($1 - \text{instability}$).
Timeline illustrating the measurement of pruning instability: testing immediately before and after a pruning event.

To isolate the cause of generalization, they compared standard pruning (removing the smallest weights) against several alternatives:

  1. Pruning Random Weights ($Prune_R$): Less stable than standard pruning.
  2. Pruning Largest Weights ($Prune_L$): Highly unstable.
  3. Temporary Zeroing: Setting weights to zero for a few batches and then allowing them to "re-enter" the training process.
  4. Gaussian Noise Injection: Adding noise to weights instead of removing them.

Key Insights: The Stability Tradeoff

The study reveals a clear negative correlation between stability and final accuracy. Pruning methods that are "gentle" (high stability) often result in lower final generalization than methods that are "harsh" (low stability).

Training curves showing the accuracy drop and recovery after pruning, alongside scatter plots demonstrating the negative correlation between stability and final test accuracy.
  • Pruning as Noise Injection: Pruning acts like a massive, structured burst of noise. The authors show that you don't even need to permanently remove parameters to get the generalization benefit; simply zeroing them out temporarily (weight re-entry) produces nearly identical results.
  • The Flatness Connection: Unstable pruning pushes the optimizer to find "flatter" minima. By measuring approximations of the Hessian (curvature) and gradient covariance, the authors confirmed that lower pruning stability correlates with flatter loss landscapes. In these regions, the model's predictions are less sensitive to small changes in weights or input data, a hallmark of good generalization.
  • Resolving the Paradox: This dynamic explains why both overparameterized networks and pruned networks generalize well: both tend to converge to flat minima. Pruning is simply a "noisy" path to get there.

Notable Results

  • Strong Correlations: In VGG11, the authors found a strong Pearson correlation ($r = -0.73$) between mean stability and final test accuracy.
  • Pruning Large Weights: Counter-intuitively, pruning the largest magnitude weights (usually considered the most "important") often led to better final generalization than pruning the smallest weights, provided the network was given time to recover.
  • Remarkable Recovery: Even after dramatic drops in accuracy (e.g., a 14% drop in a single pruning step), networks often recovered to a higher accuracy than their more stable counterparts within a few epochs.

Limitations and Future Work

While instability is beneficial, the tradeoff isn't infinite. If pruning is too unstable—such as pruning too many large weights at once—the network fails to recover, and generalization collapses. Finding the optimal "dose" of instability remains an open challenge.

Furthermore, while the authors tested several ResNet variants and VGG, the study focuses on convolutional neural networks (CNNs). It remains to be seen if these stability-generalization dynamics behave the same way in Transformers or Large Language Models. Finally, the strength of the tradeoff varies based on the iterative pruning rate and the specific layers targeted, suggesting that pruning schedules must be carefully tuned to maximize this regularization effect.

Summary updated

Abstract

Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes massive reductions in parameter counts. To better understand this phenomenon, we analyze the behavior of pruning over the course of training, finding that pruning's benefit to generalization increases with pruning's instability (defined as the drop in test accuracy immediately following pruning). We demonstrate that this "generalization-stability tradeoff'' is present across a wide variety of pruning settings and propose a mechanism for its cause: pruning regularizes similarly to noise injection. Supporting this, we find less pruning stability leads to more model flatness and the benefits of pruning do not depend on permanent parameter removal. These results explain the compatibility of pruning-based generalization improvements and the high generalization recently observed in overparameterized networks.
BibTeX
@inproceedings{bartoldson2020generalization,
  author = {Brian R. Bartoldson and Ari S. Morcos and Adrian Barbu and Gordon Erlebacher},
  title = {{The Generalization-Stability Tradeoff In Neural Network Pruning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=eSHJaceb4fw},
}
Figure