TL;DR
This paper introduces a novel "polarization regularizer" for structured neural network pruning that, unlike traditional L1 regularization, explicitly pushes neuron scaling factors toward two distinct poles: zero (to be pruned) or a fixed positive value (to be kept). By creating a clear margin between important and unimportant neurons, the method achieves state-of-the-art results, often reducing FLOPs by over 50% with negligible accuracy loss—and in some cases, even improving performance.
What Problem Does This Paper Address?
Deep learning models are notoriously resource-hungry. To deploy them on mobile devices or edge hardware, researchers use pruning to remove redundant parameters. There are two main flavors:
- Unstructured pruning: Removes individual weights, leading to sparse matrices that require specialized hardware to actually run faster.
- Structured pruning: Removes entire neurons or channels. This results in smaller, standard matrices that provide immediate speedups on any CPU or GPU.
The standard way to perform structured pruning is to attach a "scaling factor" to each neuron (often reusing the $\gamma$ parameter from Batch Normalization) and apply L1 regularization. The idea is to push these factors toward zero and then "cut" the neurons below a certain threshold.
The Gap: L1 regularization is "indiscriminate"—it tries to push every scaling factor toward zero. This often results in a dense cluster of neurons near the pruning threshold, making it difficult to decide where to cut without causing a massive drop in accuracy. The authors argue that a better approach should simultaneously suppress unimportant neurons while keeping important ones "healthy" and far from the chopping block.
Methods
The authors propose the Polarization Regularizer. Mathematically, it modifies the standard sparsity objective by adding a term that pushes scaling factors away from their mean:
$$R_s(\gamma) = t||\gamma||_1 - ||\gamma - \bar{\gamma}\mathbf{1}_n||_1$$
- The first term ($||\gamma||_1$) is the standard L1 penalty that encourages sparsity.
- The second term ($||\gamma - \bar{\gamma}\mathbf{1}_n||_1$) is the "polarization" part. It measures how far each scaling factor is from the average. By subtracting this (minimizing the negative), the optimizer is forced to push factors away from the average and toward the extremes (0 or a target value $a$).
Key Technical Components: * Theoretical Guarantee: The authors prove that the global minimum of this regularizer indeed pushes a specific proportion of neurons to zero and the rest to a fixed value $a$. * Automatic Thresholding: Because the regularizer creates a "no-man's land" (a margin) between pruned and kept neurons, the authors can automatically find the pruning threshold by looking for the first local minimum in the distribution histogram. * Hyperparameter Search: They provide a dual-stage search algorithm to find the best values for $\lambda$ (regularization strength) and $t$ (which controls the pruning ratio) to hit a specific target for FLOPs reduction.
Key Insights and Ideas
- The Power of the Margin: The most striking insight is visual. In traditional L1 pruning, the distribution of scaling factors is a continuous slope. In Polarization pruning, the distribution looks like two distinct islands. This separation makes the model much more robust to the actual pruning step.
- Reusing Batch Normalization: By applying the regularizer to the existing scaling factors in Batch Normalization (BN) layers, the method requires almost no architectural changes to existing models.
- "Healthy" Neurons: By pushing the "kept" neurons toward a specific positive value, the regularizer ensures that the remaining sub-network is well-scaled and ready for fine-tuning, rather than being half-suppressed by L1.
Key Results
The "Polarization" method consistently outperformed existing state-of-the-art (SOTA) pruning techniques like Network Slimming (NS) and AutoPruner across standard image classification benchmarks (CIFAR-10, CIFAR-100, and ImageNet).
- ImageNet (ResNet-50): Reduced FLOPs by 54% with only a 0.52% drop in top-1 accuracy. For comparison, other methods often see drops of 1-2% for similar reduction levels.
- MobileNet V2: Even on this already-compact architecture, they achieved a 28% FLOPs reduction with a tiny 0.2% accuracy drop, outperforming MetaPruning and AMC.
- CIFAR-10 (ResNet-56): Achieved a 47% FLOPs reduction while actually increasing accuracy by 0.03% over the baseline.
- Extreme Pruning: In high-compression scenarios (70%+ FLOPs reduction), the polarization method maintained significantly higher accuracy than L1-based methods, proving the benefit of the "margin" in high-stress regimes.
Limitations and Future Work
- Search Overhead: While the authors claim the search for hyperparameters $\lambda$ and $t$ isn't too expensive (requiring about 5 short training runs), it still adds a layer of complexity compared to simple one-shot pruning.
- Hyperparameter Sensitivity: Although they find the results are "insensitive" to the exact value of the upper bound $a$, the balance between $t$ and $\lambda$ is crucial for hitting specific hardware targets.
- Future Directions: The authors suggest that the "polarization effect" could be useful in other areas of deep learning beyond pruning, such as quantization or neural architecture search (NAS), where making discrete "stay or go" decisions is a common challenge.
Summary updated