Introduction
Deep Neural Networks (DNNs) are often excessively over-parameterized, posing significant challenges for deployment on resource-constrained edge devices. Pruning—the process of removing redundant weights—is a common strategy to mitigate this. Specifically, structured pruning (or channel pruning) is preferred for hardware deployment as it removes entire filters, allowing for immediate speedups on standard hardware without specialized libraries.
A major limitation in current state-of-the-art pruning methods is the reliance on a global threshold. By applying a single threshold across the entire network, these methods often ignore the unique weight distributions of individual layers. This "one-size-fits-all" approach frequently leads to "over-pruning" in sensitive layers, which can catastrophically degrade model accuracy, or "under-pruning" in others, leaving redundant parameters intact.
Methodology: Optimal Thresholding (OT)
Researchers at Intel have introduced Optimal Thresholding (OT), a method that replaces rigid global thresholds with flexible, layer-dependent ones. The OT pipeline consists of three main stages:
- Sparsity-Induced Training: The network is trained using $L_1$ regularization on the scaling factors ($\gamma$) of Batch Normalization (BN) layers. This process effectively pushes unimportant channels toward zero.
- Optimal Thresholding: After sparsity training, the scaling factors typically exhibit a bimodal distribution. One group of values clusters near zero (negligible), while another group remains significantly larger (important). OT calculates a unique threshold for each layer by identifying the "gap" between these two populations. Specifically, it finds the point where the cumulative sum of squares of the scaling factors reaches a predefined ratio ($\delta$), separating signal from noise on a per-layer basis.
- Fine-tuning: The resulting compact model is fine-tuned to recover any minor accuracy loss.
Experimental Results
The OT method was rigorously validated on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet (ILSVRC-2012) using architectures like VGG-14/16, ResNet-50, and DenseNet-121.
Performance and Sparsity
OT consistently outperforms "Network Slimming" (NS), the standard global thresholding approach. As shown in the experimental results, OT maintains high accuracy even at extreme levels of FLOP and parameter reduction, whereas global methods often see an accuracy "crash" at high sparsity. For instance, a pruned DenseNet-121 on CIFAR-100 achieved 75.99% accuracy with a massive reduction to only 0.74M parameters.
Hardware Acceleration
Beyond theoretical metrics, the authors demonstrated real-world performance gains on the Intel Neural Compute Stick 2 (NCS 2). The pruned models achieved a 2.48x speedup with zero accuracy loss and up to a 4.25x speedup with less than 5% relative accuracy loss, proving the practical utility of the OT approach for edge deployment.
Key Insights
- The Bimodal Gap: The success of OT stems from the realization that $L_1$ regularization naturally splits weights into two distinct populations. The "gap" between them provides a mathematically sound location for pruning.
- Layer Independence: By treating each layer's distribution as unique, OT prevents the accidental deletion of critical layers, ensuring stable information flow through the network.
- Rapid Convergence: Due to the precision of the initial cut, OT-pruned models require significantly less fine-tuning—often converging in just a few iterations compared to the dozens of epochs required by traditional methods.
Conclusion
Optimal Thresholding provides a robust and efficient framework for channel pruning. By leveraging the natural bimodal distribution of weights after sparsity training, it achieves superior compression ratios and hardware speedups while maintaining model performance. While currently dependent on Batch Normalization, the method's stability and one-shot pruning capability make it a powerful tool for optimizing modern vision architectures.
Summary updated