Magnitude and Uncertainty Pruning Criterion for Neural Networks

Vinnie Ko, Stefan Oehmcke, Fabian Gieseke

Abstract

Modern deep learning models are often overparameterized, leading to high memory and computational costs. While pruning is a common solution, traditional magnitude-based methods are sensitive to weight scaling and training noise. We introduce the Magnitude and Uncertainty (M&U) pruning criterion, which incorporates a "Pseudo Bootstrap" method to estimate weight stability. Our approach is scale-invariant and consistently outperforms standard magnitude pruning across various benchmarks, including MNIST, CIFAR-10, and ImageNet subsets.

Introduction

Neural network pruning aims to remove redundant parameters to facilitate deployment on edge devices. The most prevalent technique, Magnitude Pruning (ABS), removes weights with the smallest absolute values. However, ABS suffers from two primary weaknesses: it is scale-sensitive (changing input units can alter pruning decisions) and it ignores weight stability. A large weight that fluctuates significantly during training may be less important than a smaller, more stable weight.

To address these issues, we propose the M&U criterion, which treats pruning as a statistical significance test, ensuring that only "certain" and large weights are retained.

The M&U Pruning Framework

The Criterion

Inspired by the Wald test, the M&U criterion ($ au_j$) evaluates each weight $w_j$ by its magnitude relative to its uncertainty:

$$\tau_j = \frac{|\hat{w}_j|}{\lambda + \tilde{\sigma}_{\hat{w}_j}}$$

where $\tilde{\sigma}_{\hat{w}_j}$ represents the estimated standard deviation (uncertainty) of the weight, and $\lambda$ is a hyperparameter balancing magnitude and stability. By dividing by the standard deviation, the criterion becomes scale-invariant.

Pseudo Bootstrap Uncertainty

Calculating true statistical uncertainty in deep networks is computationally expensive. We propose "Pseudo Bootstrap" as a lightweight alternative. During the final iterations of training (e.g., the last 200 steps of SGD), we record weight values. The standard deviation of these temporal fluctuations serves as a proxy for uncertainty, capturing how much a weight varies as the model converges.

Comparison of pruning criteria (ABS, M&U with bootstrap, and M&U with pseudo-bootstrap) on Fashion MNIST.

Experimental Results

Benchmarks and Performance

We evaluated M&U against the ABS baseline across several datasets:

  • Fashion MNIST & MNIST: M&U demonstrated superior performance, particularly at extreme pruning levels. On MNIST, the model retained high accuracy even when 99% of weights were removed, suggesting the criterion effectively identifies the core "signal" amidst overparameterized noise.
  • CIFAR-10: Using an AlexNet-style architecture, the Pseudo Bootstrap version of M&U outperformed ABS in the majority of test cases.
  • ImageNet (VGG-11): On a 100-class subset of ImageNet, M&U consistently maintained a lower accuracy drop than ABS across all pruning levels, as shown in the figure below.
Test accuracy drop vs. prune level for ABS and M&U_pb on a VGG-11 architecture (ImageNet subset).

Key Insights

  1. Scale Invariance: Unlike magnitude pruning, M&U is robust to unit transformations in the input data, making the pruning process more reliable across different data preprocessing pipelines.
  2. Regularization Effect: In highly overparameterized settings (like MLPs on MNIST), pruning with M&U actually improved test accuracy by removing unstable weights that likely contributed to overfitting.

Conclusion and Future Work

The M&U criterion provides a theoretically grounded and practically efficient alternative to magnitude-based pruning. By accounting for weight uncertainty via Pseudo Bootstrap, it achieves higher compression rates with minimal accuracy loss. Future research will focus on scaling this method to massive architectures like Transformers and combining it with other compression techniques such as quantization.

Summary updated

BibTeX
@inproceedings{ko2019magnitude,
  author = {Vinnie Ko and Stefan Oehmcke and Fabian Gieseke},
  title = {{Magnitude and Uncertainty Pruning Criterion for Neural Networks}},
  booktitle = {IEEE BigData},
  year = {2019},
  doi = {10.1109/bigdata47090.2019.9005692},
}
Figure