Exploring the Effect of ℓ0/ℓ2 Regularization in Neural Network Pruning using the LC Toolkit

Yerlan Idelbayev, Miguel Á. Carreira-Perpiñán

Abstract

The LC Toolkit is an open-source library written in Python and PyTorch that allows to compress any neural network using several compressions including quantization, pruning, and low-rank. The versatility of the framework is rooted in the principled mathematical formulation of the underlying network compression problems with subsequent optimization by learning-compression (LC) algorithm. In this paper, we utilize the LC toolkit’s common algorithmic base to take a deeper look into ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> -constrained pruning problems defined as follows: given a budget of κ non-zero weights, which weights should be pruned in the final network? We observe that ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> -pruned networks have a different connectivity structure compared to pruning results using ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> norm. We propose a change to the formulation of the problem involving a small amount of ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> weight decay which has a favorable effect on connectivity structure. We study the properties of the proposed ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</inf> + ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> formulation using the LC toolkit and empirically demonstrate that such a scheme achieves a competitive sparsity-error tradeoff while having better structural sparsity.
BibTeX
@inproceedings{idelbayev2022exploring,
  author = {Yerlan Idelbayev and Miguel Á. Carreira-Perpiñán},
  title = {{Exploring the Effect of ℓ0/ℓ2 Regularization in Neural Network Pruning using the LC Toolkit}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=tL0YRb5g7F},
}

Loading related content...

Figure