Structured Compression of Deep Neural Networks with Debiased Elastic Group LASSO

Oyebade K. Oyedotun, Djamila Aouada, Björn E. Ottersten

State-of-the-art Deep Neural Networks (DNNs) are often "cumbersome," characterized by massive memory footprints and high computational costs (FLOPs). These requirements make deployment on mobile or edge devices challenging. While unstructured compression (zeroing individual weights) can reduce model size, it requires specialized hardware to achieve actual speedups. In contrast, structured compression—removing entire filters or channels—is more practical as it results in a physically smaller, standard architecture.

However, the standard tool for structured sparsity, Group LASSO, suffers from three major flaws: saturation (limited feature selection), correlation issues (instability with similar filters), and shrinkage bias (regularization that hurts final accuracy). This paper introduces Debiased Elastic Group LASSO (DEGL) to overcome these limitations.

The DEGL Methodology

The authors propose a three-stage pipeline designed to create significantly smaller, faster models without the performance degradation typically seen in traditional LASSO-based methods.

  1. Elastic Group LASSO (EGL) Training: The cost function is modified to include both a Group LASSO penalty (for structured sparsity) and an $l_2$-norm penalty (from Elastic Net). The $l_2$ term helps handle correlated features and breaks the saturation limit, allowing for better feature selection in over-parameterized models.
  2. Pruning: After training, the importance of each filter is assessed based on its maximum absolute weight value. Filters falling below a specific threshold ($t_{th}$) are removed entirely.
  3. Debiasing: This critical final step involves retraining the pruned model without the Group LASSO penalty. By removing the sparsity constraint during retraining, the remaining weights can "un-shrink" and reach their optimal values, disentangling feature selection from the negative side effects of regularization.
The DEGL pipeline: from initial training with elastic group LASSO to the final pruned and debiased model.

Experimental Validation and Results

DEGL was validated across a wide range of standard benchmarks (MNIST, CIFAR-10, CIFAR-100, ImageNet) and domain-specific tasks (Oxford-102, Food-5K) using architectures like VGG-16, AlexNet, and ResNet.

Key results demonstrate the efficiency of the method:

  • VGG-16 on CIFAR-10: Achieved a 90.0% reduction in parameters and an 88.1% reduction in FLOPs with a test error increase of only 0.18%.
  • AlexNet on ImageNet: Pruned 67.6% of parameters with a negligible error increase of 0.31%, significantly outperforming standard Group LASSO (1.37% error increase).
  • ResNet-50 on ImageNet: DEGL pruned 54.4% of parameters with only a 0.23% error increase, outperforming other state-of-the-art pruning methods like ThiNet.
  • Domain Adaptation: On the Oxford-102 dataset, DEGL achieved 43.9% parameter reduction while actually improving the error rate by 0.01%, suggesting that the compression can act as a beneficial regularizer.

Key Insights

  • Structured is Better for Deployment: Because DEGL removes entire filters, the resulting model is simply a smaller version of the original, requiring no specialized AI chips for inference speedups.
  • The Power of Debiasing: The authors show that the "shrinkage" effect of LASSO is a significant contributor to accuracy loss. Turning off the penalty during retraining allows the model to recover its performance.
  • Memory vs. Compute: The study highlights that DEGL optimizes both the Model Memory Size (storage) and the Computational Memory Footprint (RAM used during inference).

Conclusion

DEGL provides a robust framework for structured DNN compression by addressing the inherent biases of Group LASSO. While future work may involve automating hyperparameter tuning and expanding the scope to Vision Transformers or LLMs, DEGL currently offers a highly effective path for deploying high-performance models on resource-constrained hardware.

Summary updated

Abstract

State-of-the-art Deep Neural Networks (DNNs) are typically too cumbersome to be practically useful in portable electronic devices. As such, several works pursue model compression that seeks to drastically reduce computational memory footprints, FLOPS and memory for storage. Many of these works achieve unstructured compression, where the compressed models are not directly useful since dedicated hardware and specialized algorithms are required for storage of sparse weights and fast sparse matrix-vector multi-plication respectively. In this paper, we propose structured compression of large DNNs using debiased elastic group LASSO (DEGL), which is motivated by different interesting characteristics of the individual components. That is, where group LASSO penalty enforces structured sparsity, l2-norm penalty promotes features grouping, and debiasing disentangles sparsity and shrinkage effects of group LASSO. We perform extensive experiments by applying DEGL to different DNN architectures including LeNet, VGG, AlexNet and ResNet on MNIST, CIFAR-10, CIFAR-100 and ImageNet datasets. Furthermore, we validate the effectiveness of our proposal on domain adaptation using Oxford-102 flower species and Food-5K datasets. Results show that DEGL can compress DNNs by several folds with small or no loss of performance. Particularly, DEGL outperforms conventional group LASSO and several other state-of-the-art methods that perform structured compression.
BibTeX
@inproceedings{oyedotun2020structured,
  author = {Oyebade K. Oyedotun and Djamila Aouada and Björn E. Ottersten},
  title = {{Structured Compression of Deep Neural Networks with Debiased Elastic Group LASSO}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093377},
}
Figure