TL;DR
Applying an $L_1$ penalty to BatchNorm scaling factors penalizes crucial and superfluous channels alike, leading to degraded performance when heavily pruning networks. To solve this, the authors introduce $\sigma$BatchNorm—a bounded variant of BatchNorm—paired with Receding Neuron Importance (RNI), a regularizer whose gradient decays exponentially for high-importance neurons. This produces a polarized bimodal importance distribution that preserves critical early layers, dramatically outperforming prior structured pruning methods under severe pruning regimes.
What Problem Does This Paper Address?
Structured pruning accelerates neural networks on standard hardware by removing entire channels or filters. A standard way to identify pruneable filters (e.g., Network Slimming) is to look at the learnable scaling factor $\gamma$ in BatchNorm layers: channels with near-zero scales contribute little and can be safely eliminated.
However, existing methods typically apply an $L_1$ penalty to push $\gamma$ to zero. Because the derivative of the $L_1$ norm is constant, it exerts the same shrinking force on crucial channels as it does on redundant ones, forcing the classification loss to constantly fight the regularizer. Furthermore, because standard BatchNorm scaling parameters are unbounded, setting consistent thresholds across layers to define what counts as "unimportant" is difficult.
Methods
The authors propose a two-part solution:
-
Sigmoid BatchNorm ($\sigma\text{BN}$): Standard BatchNorm computes $y = \gamma \hat{x} + \beta$. The authors show that the bias term $\beta$ can be removed without sacrificing accuracy, and they bound the scaling factor into the range $(0, 1)$ using a sigmoid function: $$\sigma\text{BN}(x, \gamma) = \sigma(\gamma) \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$ This gives every channel an interpretable, bounded importance score $I(\theta_i) = \sigma(\gamma_i) \in (0, 1)$.
-
Receding Neuron Importance (RNI) Regularization: Using the bounded parameter, the authors design a loss term: $$R_s(\gamma, b) = \sigma(\gamma + b) \cdot \bigl(1 - \log(\sigma(\gamma + b))\bigr)$$ where $b$ is a shift parameter. The gradient of $R_s$ with respect to $\gamma$ decays exponentially as $\sigma(\gamma)$ grows. As a result, the regularizer forcefully drives low-importance parameters toward zero while leaving high-importance parameters virtually untouched, naturally polarizing the weights into a bimodal distribution of "keep" versus "prune."
Pruning follows a standard one-shot pipeline: train with RNI, prune channels globally below the threshold, and fine-tune.
Datasets
The authors evaluate their method on standard computer vision benchmarks:
- CIFAR-10 and CIFAR-100 (using VGG-16 and ResNet-56)
- ImageNet (using ResNet-50)
No new datasets are introduced.
Key Insights and Ideas
- Why global pruning fails at high compression: In architectures like VGG, early layers have very few filters (e.g., 64), while deep layers have many (e.g., 512). Global pruning baselines inadvertently strip away disproportionate fractions of early layers to satisfy global targets, bottlenecking feature extraction. RNI naturally avoids over-pruning early layers because their importance scores stay high.
- Architectural resilience: ResNets degrade far more gracefully than VGG models across all methods because skip connections maintain gradient and signal flow even if an entire residual block is decimated.
- Pruned baselines beat compact designs: Heavily pruned VGG and ResNet models outperformed parameter-matched MobileNetV2 models trained from scratch, demonstrating that overparameterized training followed by structured pruning yields superior compact networks.
Key Results
- Resilience under severe pruning (CIFAR-100, VGG-16 at 90% filters pruned): While competing methods suffer catastrophic collapse—Slimming drops to 29.66%, DeepHoyer to 25.42%, and Polarization to 25.62%—RNI retains 57.92% accuracy (outperforming even the uniform channel scaling baseline of 54.39%).
- CIFAR-10 (VGG-16 at 90% filters pruned): RNI achieves 90.96% accuracy (8.40% remaining FLOPs), outperforming Slimming (89.15%) and DeepHoyer (80.82%).
- ImageNet (ResNet-50 at 90% filters pruned): RNI reaches 54.08% top-1 accuracy (17.66% remaining FLOPs), surpassing Slimming (51.74%) and DeepHoyer (50.98%), though trailing uniform local pruning (58.13%).
Limitations and Future Work
- Hyperparameter Sensitivity: The method introduces shift parameter $b$ alongside regularizer weight $\lambda_S$. At high pruning ratios, accuracy is sensitive to this tuning; over- or under-regularizing causes noticeable degradation.
- ImageNet Gap: On large-scale ImageNet benchmarks with ResNet-50, RNI was still outperformed at the 90% mark by uniform channel scaling (UCS), indicating that global ranking criteria still face hurdles on complex datasets.
- Future Directions: The authors highlight the need for automated hyperparameter scheduling that can dynamically adjust $b$ and $\lambda_S$ to target a predetermined sparsity ratio directly during training.
Summary updated