GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization

Yi Guo, Huan Yuan, Jianchao Tan, Zhangyang Wang, Sen Yang, Ji Liu

Abstract

Channel pruning is a critical technique for optimizing Deep Neural Networks (DNNs) for deployment on resource-constrained edge devices. However, traditional pruning methods often rely on heuristic importance metrics or suffer from the training instabilities inherent in sampling-based approaches. This paper presents Gates with Differentiable Polarization (GDP), a novel framework that utilizes a deterministic, differentiable gate function to drive unimportant channels toward zero. By gradually decreasing a polarization hyperparameter, GDP achieves a stable transition from a dense super-net to a sparse sub-net. Experimental results across ImageNet, CIFAR-10, and Pascal VOC demonstrate that GDP achieves state-of-the-art accuracy-to-FLOPs trade-offs without requiring extensive fine-tuning.

Introduction

Deep Neural Networks (DNNs) have achieved remarkable performance across various tasks but at a high computational cost. Channel pruning—the removal of entire filters—is a popular strategy for model compression. Existing methods typically face two hurdles: the reliance on arbitrary heuristic metrics (like weight magnitude) to determine channel importance, and the instability of stochastic sampling methods used to search for optimal sub-networks. GDP addresses these issues by introducing a differentiable polarization mechanism that deterministically identifies and suppresses redundant channels during the standard training process.

The GDP Framework

The GDP Module Architecture

The core of the proposed method is the GDP module, a simple gate vector integrated into the network architecture. These gates are placed before convolutional or fully connected layers to control the contribution of individual channels.

Integration of GDP gates ($g_1, g_2$) within a residual block.

The Polarized Gate Mechanism

Unlike standard weights, GDP employs a specific "smoothed $L_0$" function to define the gate values: $$g_{\epsilon}(x) = \frac{x^2}{x^2 + \epsilon}$$

As the hyperparameter $\epsilon$ (epsilon) is gradually decreased during training, the function creates a polarization effect. This drives channels toward two extremes: exactly zero or near one. Channels pushed to zero contribute nothing to the output and can be removed without affecting the model's performance. Conversely, channels near one remain active, and their gate values can be mathematically absorbed into the subsequent layer's weights, ensuring the final pruned model behaves identically to the trained "super-net."

Behavior of the polarized gate function $g_{\epsilon}(x)$ and its derivative $g'_{\epsilon}(x)$ for varying $\epsilon$.

Optimization via Alternative Relaxation

To balance accuracy with resource constraints (FLOPs), GDP utilizes Alternative Relaxation. The training process alternates between updating standard network weights and updating gate parameters using a proximal-SGD approach. This ensures that sparsity—the number of zeroed-out channels—grows stably and predictably throughout training, avoiding the "shocks" to the network often seen in threshold-based pruning.

Experimental Evaluation

GDP was evaluated on standard benchmarks including CIFAR-10, ImageNet (ILSVRC-12), and Pascal VOC. Key findings include:

  • ImageNet Performance: On MobileNet-V1, GDP achieved 71.3% Top-1 accuracy with significant FLOPs reduction, outperforming competitors like AutoSlim. On ResNet-50, it maintained high accuracy while reducing FLOPs by over 50%.
  • Semantic Segmentation: Tested on DeepLabV3+ (Pascal VOC), GDP achieved a 60% FLOPs saving while slightly increasing the Mean Intersection over Union (mIoU) from 0.774 to 0.779.
  • Consistency and Efficiency: A major advantage of GDP is that the pruned model's performance is identical to the model before physical removal of channels. This consistency eliminates the need for the traditional, time-consuming "prune-then-fine-tune" cycle.

Conclusion

GDP provides a robust and differentiable path to neural network pruning. By replacing heuristics and random sampling with a polarized gating mechanism, it offers a stable way to achieve state-of-the-art compression. While the pruning ratio currently requires tuning via a balance factor ($\lambda$), the method's robustness to other hyperparameters makes it a practical tool for model optimization. Future work may extend this differentiable polarization to the broader field of Neural Architecture Search (NAS).

Summary updated

Abstract

Model compression techniques are recently gaining explosive attention for obtaining efficient AI models for various real-time applications. Channel pruning is one important compression strategy and is widely used in slimming various DNNs. Previous gate-based or importance-based pruning methods aim to remove channels whose importance is smallest. However, it remains unclear what criteria the channel importance should be measured on, leading to various channel selection heuristics. Some other sampling-based pruning methods deploy sampling strategies to train sub-nets, which often causes the training instability and the compressed model's degraded performance. In view of the research gaps, we present a new module named Gates with Differentiable Polarization (GDP), inspired by principled optimization ideas. GDP can be plugged before convolutional layers without bells and whistles, to control the on-and-off of each channel or whole layer block. During the training process, the polarization effect will drive a subset of gates to smoothly decrease to exact zero, while other gates gradually stay away from zero by a large margin. When training terminates, those zero-gated channels can be painlessly removed, while other non-zero gates can be absorbed into the succeeding convolution kernel, causing completely no interruption to training nor damage to the trained model. Experiments conducted over CIFAR-10 and ImageNet datasets show that the proposed GDP algorithm achieves the state-of-the-art performance on various benchmark DNNs at a broad range of pruning ratios. We also apply GDP to DeepLabV3Plus-ResNet50 on the challenging Pascal VOC segmentation task, whose test performance sees no drop (even slightly improved) with over 60% FLOPs saving.
BibTeX
@inproceedings{guo2021gdp,
  author = {Yi Guo and Huan Yuan and Jianchao Tan and Zhangyang Wang and Sen Yang and Ji Liu},
  title = {{GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=g1Ju1JmKvIU},
}
Figure