State-of-the-art Deep Neural Networks (DNNs) are often "cumbersome," characterized by massive memory footprints and high computational costs (FLOPs). These requirements make deployment on mobile or edge devices challenging. While unstructured compression (zeroing individual weights) can reduce model size, it requires specialized hardware to achieve actual speedups. In contrast, structured compression—removing entire filters or channels—is more practical as it results in a physically smaller, standard architecture.
However, the standard tool for structured sparsity, Group LASSO, suffers from three major flaws: saturation (limited feature selection), correlation issues (instability with similar filters), and shrinkage bias (regularization that hurts final accuracy). This paper introduces Debiased Elastic Group LASSO (DEGL) to overcome these limitations.
The DEGL Methodology
The authors propose a three-stage pipeline designed to create significantly smaller, faster models without the performance degradation typically seen in traditional LASSO-based methods.
- Elastic Group LASSO (EGL) Training: The cost function is modified to include both a Group LASSO penalty (for structured sparsity) and an $l_2$-norm penalty (from Elastic Net). The $l_2$ term helps handle correlated features and breaks the saturation limit, allowing for better feature selection in over-parameterized models.
- Pruning: After training, the importance of each filter is assessed based on its maximum absolute weight value. Filters falling below a specific threshold ($t_{th}$) are removed entirely.
- Debiasing: This critical final step involves retraining the pruned model without the Group LASSO penalty. By removing the sparsity constraint during retraining, the remaining weights can "un-shrink" and reach their optimal values, disentangling feature selection from the negative side effects of regularization.
Experimental Validation and Results
DEGL was validated across a wide range of standard benchmarks (MNIST, CIFAR-10, CIFAR-100, ImageNet) and domain-specific tasks (Oxford-102, Food-5K) using architectures like VGG-16, AlexNet, and ResNet.
Key results demonstrate the efficiency of the method:
- VGG-16 on CIFAR-10: Achieved a 90.0% reduction in parameters and an 88.1% reduction in FLOPs with a test error increase of only 0.18%.
- AlexNet on ImageNet: Pruned 67.6% of parameters with a negligible error increase of 0.31%, significantly outperforming standard Group LASSO (1.37% error increase).
- ResNet-50 on ImageNet: DEGL pruned 54.4% of parameters with only a 0.23% error increase, outperforming other state-of-the-art pruning methods like ThiNet.
- Domain Adaptation: On the Oxford-102 dataset, DEGL achieved 43.9% parameter reduction while actually improving the error rate by 0.01%, suggesting that the compression can act as a beneficial regularizer.
Key Insights
- Structured is Better for Deployment: Because DEGL removes entire filters, the resulting model is simply a smaller version of the original, requiring no specialized AI chips for inference speedups.
- The Power of Debiasing: The authors show that the "shrinkage" effect of LASSO is a significant contributor to accuracy loss. Turning off the penalty during retraining allows the model to recover its performance.
- Memory vs. Compute: The study highlights that DEGL optimizes both the Model Memory Size (storage) and the Computational Memory Footprint (RAM used during inference).
Conclusion
DEGL provides a robust framework for structured DNN compression by addressing the inherent biases of Group LASSO. While future work may involve automating hyperparameter tuning and expanding the scope to Vision Transformers or LLMs, DEGL currently offers a highly effective path for deploying high-performance models on resource-constrained hardware.
Summary updated