Transformed 𝓁1 Regularization for Learning Sparse Deep Neural Networks

Rongrong Ma, Jianyu Miao, Lingfeng Niu, Peng ZhangNeural Networks 2019 · OpenReview

TL;DR

This paper introduces a novel non-convex regularization technique called "integrated transformed ℓ₁" (ITL) to create sparser and more efficient deep neural networks. By combining transformed ℓ₁ for connection sparsity and group sparsity for neuron removal, ITL achieves state-of-the-art results in reducing model size and computation while maintaining high accuracy across various datasets.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have achieved remarkable success, but their large number of parameters leads to significant memory and computational costs, making them unsuitable for resource-constrained platforms. Overfitting is also a concern with insufficient data. While many methods aim to reduce DNN complexity, they often rely on manual tuning or achieve sparsity at the expense of accuracy. This paper addresses the need for an automated and effective method to simultaneously reduce both redundant connections and unnecessary neurons in DNNs without sacrificing performance.

Methods

The core of this work is the introduction of the integrated transformed ℓ₁ (ITL) regularizer. This regularizer combines two components:

  • Transformed ℓ₁ (Tℓ₁): A non-convex regularizer that promotes sparsity at the connection level. It's a smooth, single-parameter family of functions that can interpolate between β„“β‚€ and ℓ₁ norms, offering flexibility in controlling sparsity.
  • Group Sparsity: This component promotes sparsity at the neuron level by encouraging entire groups of weights (corresponding to neurons) to be zero.

The authors propose a stochastic proximal gradient descent algorithm to efficiently train DNNs with the ITL regularizer. This algorithm iteratively updates weights layer by layer, leveraging closed-form solutions for the proximal operators of both Tℓ₁ and group sparsity, even though Tℓ₁ is non-convex.

Datasets

The authors evaluate their proposed method on six public classification datasets:

  • DIGITS: A small toy dataset of handwritten digits (1,797 8x8 grayscale images). Used to illustrate the effect of the parameter 'a' in Tℓ₁ and compare sparsity-promoting capacities.
  • MNIST: A standard dataset of handwritten digits (70,000 28x28 grayscale images).
  • Fashion-MNIST: A dataset of clothing images (60,000 training, 10,000 test, 28x28 grayscale).
  • PENDIGITS: A dataset of handwritten digits (10,992 4x4 grayscale images).
  • Sensorless Drive Diagnosis (SDD): A dataset from the UCI repository with 58,508 examples and 48 features for motor defect prediction.
  • CIFAR-10: A dataset of 32x32 color images in 10 classes (60,000 images total).

Key Insights and Ideas

  • Non-convexity for better sparsity: The paper argues that non-convex regularizers are more effective than convex ones for promoting sparsity, leading to sparser solutions and potentially better performance.
  • Dual sparsity promotion: By combining connection-level sparsity (Tℓ₁) and neuron-level sparsity (group sparsity), ITL offers a more comprehensive approach to model compression.
  • Tunable sparsity: The transformed ℓ₁ regularizer's parameter 'a' allows for flexible control over the degree of sparsity, enabling a trade-off between accuracy and model size.
  • Efficient optimization: The development of a proximal gradient algorithm with closed-form solutions for the non-smooth and non-convex regularizers is crucial for practical application.

Key Results

  • State-of-the-art performance: The ITL regularizer achieved comparable or better prediction accuracy than baselines (ℓ₁, SGL, CGES) across multiple datasets.
  • Significant reduction in FLOPs and parameters: ITL demonstrated substantial reductions in floating-point operations (FLOPs) and the number of parameters used in the network. For example, on MNIST, FLOPs were reduced by nearly 49% and parameters by 16.01% compared to baselines, while maintaining high accuracy.
  • Improved convergence speed: Networks trained with ITL converged faster than those trained with other regularizers on the MNIST dataset.
  • Effective filter pruning: Visualization of filters in convolutional layers showed that ITL effectively removed redundant features, resulting in sharper and more efficient filters compared to other methods.
  • Parameter 'a' impact: Experiments on the DIGITS dataset showed that the choice of parameter 'a' in Tℓ₁ significantly impacts accuracy, FLOPs, and parameter usage, with values around 10⁻² to 10⁻¹ often yielding good results.

Limitations and Future Work

  • Limited architecture tested: The current work primarily verifies the effectiveness of ITL on convolutional neural networks.
  • Future exploration: The authors plan to test ITL on other neural network architectures. They also aim to explore replacing the β„“β‚‚ norm in group sparsity with other regularizers to further enhance the single regularizer's ability to remove both redundant neurons and connections simultaneously.

Summary updated

Abstract

Deep neural networks (DNNs) have achieved extraordinary success in numerous areas. However, to attain this success, DNNs often carry a large number of weight parameters, leading to heavy costs of memory and computation resources. Overfitting is also likely to happen in such network when the training data are insufficient. These shortcomings severely hinder the application of DNNs in resource-constrained platforms. In fact, many network weights are known to be redundant and can be removed from the network without much loss of performance. To this end, we introduce a new non-convex integrated transformed $\ell_1$ regularizer to promote sparsity for DNNs, which removes both redundant connections and unnecessary neurons simultaneously. To be specific, we apply the transformed $\ell_1$ to the matrix space of network weights and utilize it to remove redundant connections. Besides, group sparsity is also employed as an auxiliary to remove unnecessary neurons. An efficient stochastic proximal gradient algorithm is presented to solve the new model at the same time. To the best of our knowledge, this is the first work to utilize a non-convex regularizer in sparse optimization based method to promote sparsity for DNNs. Experiments on several public datasets demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{ma2019transformed,
  author = {Rongrong Ma and Jianyu Miao and Lingfeng Niu and Peng Zhang},
  title = {{Transformed 𝓁1 Regularization for Learning Sparse Deep Neural Networks}},
  booktitle = {Neural Networks},
  year = {2019},
  url = {https://openreview.net/forum?id=BHgDwlkwdVx},
}
Figure