Receding Neuron Importances for Structured Pruning

Mihai Suteu, Yike Guo

TL;DR

Applying an $L_1$ penalty to BatchNorm scaling factors penalizes crucial and superfluous channels alike, leading to degraded performance when heavily pruning networks. To solve this, the authors introduce $\sigma$BatchNorm—a bounded variant of BatchNorm—paired with Receding Neuron Importance (RNI), a regularizer whose gradient decays exponentially for high-importance neurons. This produces a polarized bimodal importance distribution that preserves critical early layers, dramatically outperforming prior structured pruning methods under severe pruning regimes.

What Problem Does This Paper Address?

Structured pruning accelerates neural networks on standard hardware by removing entire channels or filters. A standard way to identify pruneable filters (e.g., Network Slimming) is to look at the learnable scaling factor $\gamma$ in BatchNorm layers: channels with near-zero scales contribute little and can be safely eliminated.

However, existing methods typically apply an $L_1$ penalty to push $\gamma$ to zero. Because the derivative of the $L_1$ norm is constant, it exerts the same shrinking force on crucial channels as it does on redundant ones, forcing the classification loss to constantly fight the regularizer. Furthermore, because standard BatchNorm scaling parameters are unbounded, setting consistent thresholds across layers to define what counts as "unimportant" is difficult.

Methods

The authors propose a two-part solution:

  1. Sigmoid BatchNorm ($\sigma\text{BN}$): Standard BatchNorm computes $y = \gamma \hat{x} + \beta$. The authors show that the bias term $\beta$ can be removed without sacrificing accuracy, and they bound the scaling factor into the range $(0, 1)$ using a sigmoid function: $$\sigma\text{BN}(x, \gamma) = \sigma(\gamma) \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$ This gives every channel an interpretable, bounded importance score $I(\theta_i) = \sigma(\gamma_i) \in (0, 1)$.

  2. Receding Neuron Importance (RNI) Regularization: Using the bounded parameter, the authors design a loss term: $$R_s(\gamma, b) = \sigma(\gamma + b) \cdot \bigl(1 - \log(\sigma(\gamma + b))\bigr)$$ where $b$ is a shift parameter. The gradient of $R_s$ with respect to $\gamma$ decays exponentially as $\sigma(\gamma)$ grows. As a result, the regularizer forcefully drives low-importance parameters toward zero while leaving high-importance parameters virtually untouched, naturally polarizing the weights into a bimodal distribution of "keep" versus "prune."

Pruning follows a standard one-shot pipeline: train with RNI, prune channels globally below the threshold, and fine-tune.

Datasets

The authors evaluate their method on standard computer vision benchmarks:

  • CIFAR-10 and CIFAR-100 (using VGG-16 and ResNet-56)
  • ImageNet (using ResNet-50)

No new datasets are introduced.

Key Insights and Ideas

  • Why global pruning fails at high compression: In architectures like VGG, early layers have very few filters (e.g., 64), while deep layers have many (e.g., 512). Global pruning baselines inadvertently strip away disproportionate fractions of early layers to satisfy global targets, bottlenecking feature extraction. RNI naturally avoids over-pruning early layers because their importance scores stay high.
  • Architectural resilience: ResNets degrade far more gracefully than VGG models across all methods because skip connections maintain gradient and signal flow even if an entire residual block is decimated.
  • Pruned baselines beat compact designs: Heavily pruned VGG and ResNet models outperformed parameter-matched MobileNetV2 models trained from scratch, demonstrating that overparameterized training followed by structured pruning yields superior compact networks.

Key Results

  • Resilience under severe pruning (CIFAR-100, VGG-16 at 90% filters pruned): While competing methods suffer catastrophic collapse—Slimming drops to 29.66%, DeepHoyer to 25.42%, and Polarization to 25.62%—RNI retains 57.92% accuracy (outperforming even the uniform channel scaling baseline of 54.39%).
  • CIFAR-10 (VGG-16 at 90% filters pruned): RNI achieves 90.96% accuracy (8.40% remaining FLOPs), outperforming Slimming (89.15%) and DeepHoyer (80.82%).
  • ImageNet (ResNet-50 at 90% filters pruned): RNI reaches 54.08% top-1 accuracy (17.66% remaining FLOPs), surpassing Slimming (51.74%) and DeepHoyer (50.98%), though trailing uniform local pruning (58.13%).

Limitations and Future Work

  • Hyperparameter Sensitivity: The method introduces shift parameter $b$ alongside regularizer weight $\lambda_S$. At high pruning ratios, accuracy is sensitive to this tuning; over- or under-regularizing causes noticeable degradation.
  • ImageNet Gap: On large-scale ImageNet benchmarks with ResNet-50, RNI was still outperformed at the 90% mark by uniform channel scaling (UCS), indicating that global ranking criteria still face hurdles on complex datasets.
  • Future Directions: The authors highlight the need for automated hyperparameter scheduling that can dynamically adjust $b$ and $\lambda_S$ to target a predetermined sparsity ratio directly during training.

Summary updated

Abstract

Structured pruning efficiently compresses networks by identifying and removing unimportant neurons. While this can be elegantly achieved by applying sparsity-inducing regularisation on BatchNorm parameters, an L1 penalty would shrink all scaling factors rather than just those of superfluous neurons. To tackle this issue, we introduce a simple BatchNorm variation with bounded scaling parameters, based on which we design a novel regularisation term that suppresses only neurons with low importance. Under our method, the weights of unnecessary neurons effectively recede, producing a polarised bimodal distribution of importances. We show that neural networks trained this way can be pruned to a larger extent and with less deterioration. We one-shot prune VGG and ResNet architectures at different ratios on CIFAR and ImagenNet datasets. In the case of VGG-style networks, our method significantly outperforms existing approaches particularly under severe pruning.
BibTeX
@inproceedings{suteu2023receding,
  author = {Mihai Suteu and Yike Guo},
  title = {{Receding Neuron Importances for Structured Pruning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=gQsRPozZYIQ},
}

Loading related content...

Figure