Receding Neuron Importances for Structured Pruning

Mihai Suteu, Yike Guo

Abstract

Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under a severe pruning regime.

Introduction: The Challenge of Structured Pruning

Deep neural networks are notoriously over-parametrized, containing far more neurons than necessary for their target tasks. Structured pruning aims to identify and remove these redundant neurons to facilitate deployment on resource-constrained hardware. A common approach involves applying sparsity-inducing regularization, such as an L1 penalty, to the scaling factors ($\gamma$) of BatchNorm layers.

Density distributions of standard BatchNorm parameters $\gamma$ and $\beta$ for VGG-19 and ResNet-50.

However, standard penalties are "blind" to neuron importance. They apply a constant shrinking pressure regardless of whether a neuron is vital to the network's logic or entirely redundant. This often results in a "mushy" distribution of weights, making it difficult to distinguish which neurons to prune. Furthermore, it can inadvertently suppress critical early layers, leading to significant performance degradation.

Methodology: Sigmoid BatchNorm and Receding Penalties

To address these limitations, we introduce two key innovations: Sigmoid BatchNorm ($\sigma$BN) and Receding Neuron Importance (RNI) regularization.

In standard BatchNorm, the scaling parameter $\gamma$ can take any value. We replace this with $\sigma$BN, where $\gamma$ is passed through a sigmoid function, bounding the "importance score" strictly between 0 and 1. We also find the learnable offset $\beta$ to be redundant for pruning purposes and remove it.

The RNI penalty gradient (left) and the resulting bimodal distribution of neuron importances compared to L1 regularization (right).

The RNI regularization term is designed such that the "pressure" to shrink a neuron decays exponentially as its importance increases. As shown in the gradient plot above, if a neuron's score is low, the penalty is high. As the neuron "proves its worth" and its score increases, the penalty "recedes," leaving important neurons untouched. This mechanism forces the network to make a binary choice, creating a polarized bimodal distribution where neurons are either very close to 1 (important) or very close to 0 (prunable).

Experimental Results and Polarization

The advantages of RNI are most pronounced under severe pruning regimes. For instance, when removing 90% of filters from a VGG-16 model on CIFAR-100, RNI maintained an accuracy of 57.92%. In contrast, traditional methods like "Network Slimming" (L1) saw performance plummet to approximately 25-30%.

On ResNet-56 for CIFAR-10, RNI achieved 90.26% accuracy at 90% pruning, outperforming all other regularized methods. Our results indicate that the bimodal distribution produced by RNI allows for a much cleaner pruning step, as there is no long tail of semi-important neurons to complicate the removal process. Furthermore, RNI naturally protects the "roots" of the network—the early layers that extract basic features—as these neurons typically maintain high importance scores and thus avoid suppression.

Conclusion

Receding Neuron Importance (RNI) offers a powerful alternative to traditional L1-based structured pruning. By bounding importance scores and applying a receding penalty, we achieve a polarized distribution of neurons that facilitates aggressive pruning with minimal accuracy loss. This method proves particularly effective for VGG-style architectures and severe compression ratios, often outperforming models designed from scratch for efficiency, such as MobileNetV2.

Summary updated

Abstract

Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under a severe pruning regime.
BibTeX
@inproceedings{suteu2023receding,
  author = {Mihai Suteu and Yike Guo},
  title = {{Receding Neuron Importances for Structured Pruning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=gQsRPozZYIQ},
}
Figure