Abstract
Channel pruning is a critical technique for optimizing Deep Neural Networks (DNNs) for deployment on resource-constrained edge devices. However, traditional pruning methods often rely on heuristic importance metrics or suffer from the training instabilities inherent in sampling-based approaches. This paper presents Gates with Differentiable Polarization (GDP), a novel framework that utilizes a deterministic, differentiable gate function to drive unimportant channels toward zero. By gradually decreasing a polarization hyperparameter, GDP achieves a stable transition from a dense super-net to a sparse sub-net. Experimental results across ImageNet, CIFAR-10, and Pascal VOC demonstrate that GDP achieves state-of-the-art accuracy-to-FLOPs trade-offs without requiring extensive fine-tuning.
Introduction
Deep Neural Networks (DNNs) have achieved remarkable performance across various tasks but at a high computational cost. Channel pruning—the removal of entire filters—is a popular strategy for model compression. Existing methods typically face two hurdles: the reliance on arbitrary heuristic metrics (like weight magnitude) to determine channel importance, and the instability of stochastic sampling methods used to search for optimal sub-networks. GDP addresses these issues by introducing a differentiable polarization mechanism that deterministically identifies and suppresses redundant channels during the standard training process.
The GDP Framework
The GDP Module Architecture
The core of the proposed method is the GDP module, a simple gate vector integrated into the network architecture. These gates are placed before convolutional or fully connected layers to control the contribution of individual channels.
The Polarized Gate Mechanism
Unlike standard weights, GDP employs a specific "smoothed $L_0$" function to define the gate values: $$g_{\epsilon}(x) = \frac{x^2}{x^2 + \epsilon}$$
As the hyperparameter $\epsilon$ (epsilon) is gradually decreased during training, the function creates a polarization effect. This drives channels toward two extremes: exactly zero or near one. Channels pushed to zero contribute nothing to the output and can be removed without affecting the model's performance. Conversely, channels near one remain active, and their gate values can be mathematically absorbed into the subsequent layer's weights, ensuring the final pruned model behaves identically to the trained "super-net."
Optimization via Alternative Relaxation
To balance accuracy with resource constraints (FLOPs), GDP utilizes Alternative Relaxation. The training process alternates between updating standard network weights and updating gate parameters using a proximal-SGD approach. This ensures that sparsity—the number of zeroed-out channels—grows stably and predictably throughout training, avoiding the "shocks" to the network often seen in threshold-based pruning.
Experimental Evaluation
GDP was evaluated on standard benchmarks including CIFAR-10, ImageNet (ILSVRC-12), and Pascal VOC. Key findings include:
- ImageNet Performance: On MobileNet-V1, GDP achieved 71.3% Top-1 accuracy with significant FLOPs reduction, outperforming competitors like AutoSlim. On ResNet-50, it maintained high accuracy while reducing FLOPs by over 50%.
- Semantic Segmentation: Tested on DeepLabV3+ (Pascal VOC), GDP achieved a 60% FLOPs saving while slightly increasing the Mean Intersection over Union (mIoU) from 0.774 to 0.779.
- Consistency and Efficiency: A major advantage of GDP is that the pruned model's performance is identical to the model before physical removal of channels. This consistency eliminates the need for the traditional, time-consuming "prune-then-fine-tune" cycle.
Conclusion
GDP provides a robust and differentiable path to neural network pruning. By replacing heuristics and random sampling with a polarized gating mechanism, it offers a stable way to achieve state-of-the-art compression. While the pruning ratio currently requires tuning via a balance factor ($\lambda$), the method's robustness to other hyperparameters makes it a practical tool for model optimization. Future work may extend this differentiable polarization to the broader field of Neural Architecture Search (NAS).
Summary updated