TL;DR
This paper introduces a novel non-convex regularization technique called "integrated transformed ββ" (ITL) to create sparser and more efficient deep neural networks. By combining transformed ββ for connection sparsity and group sparsity for neuron removal, ITL achieves state-of-the-art results in reducing model size and computation while maintaining high accuracy across various datasets.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) have achieved remarkable success, but their large number of parameters leads to significant memory and computational costs, making them unsuitable for resource-constrained platforms. Overfitting is also a concern with insufficient data. While many methods aim to reduce DNN complexity, they often rely on manual tuning or achieve sparsity at the expense of accuracy. This paper addresses the need for an automated and effective method to simultaneously reduce both redundant connections and unnecessary neurons in DNNs without sacrificing performance.
Methods
The core of this work is the introduction of the integrated transformed ββ (ITL) regularizer. This regularizer combines two components:
- Transformed ββ (Tββ): A non-convex regularizer that promotes sparsity at the connection level. It's a smooth, single-parameter family of functions that can interpolate between ββ and ββ norms, offering flexibility in controlling sparsity.
- Group Sparsity: This component promotes sparsity at the neuron level by encouraging entire groups of weights (corresponding to neurons) to be zero.
The authors propose a stochastic proximal gradient descent algorithm to efficiently train DNNs with the ITL regularizer. This algorithm iteratively updates weights layer by layer, leveraging closed-form solutions for the proximal operators of both Tββ and group sparsity, even though Tββ is non-convex.
Datasets
The authors evaluate their proposed method on six public classification datasets:
- DIGITS: A small toy dataset of handwritten digits (1,797 8x8 grayscale images). Used to illustrate the effect of the parameter 'a' in Tββ and compare sparsity-promoting capacities.
- MNIST: A standard dataset of handwritten digits (70,000 28x28 grayscale images).
- Fashion-MNIST: A dataset of clothing images (60,000 training, 10,000 test, 28x28 grayscale).
- PENDIGITS: A dataset of handwritten digits (10,992 4x4 grayscale images).
- Sensorless Drive Diagnosis (SDD): A dataset from the UCI repository with 58,508 examples and 48 features for motor defect prediction.
- CIFAR-10: A dataset of 32x32 color images in 10 classes (60,000 images total).
Key Insights and Ideas
- Non-convexity for better sparsity: The paper argues that non-convex regularizers are more effective than convex ones for promoting sparsity, leading to sparser solutions and potentially better performance.
- Dual sparsity promotion: By combining connection-level sparsity (Tββ) and neuron-level sparsity (group sparsity), ITL offers a more comprehensive approach to model compression.
- Tunable sparsity: The transformed ββ regularizer's parameter 'a' allows for flexible control over the degree of sparsity, enabling a trade-off between accuracy and model size.
- Efficient optimization: The development of a proximal gradient algorithm with closed-form solutions for the non-smooth and non-convex regularizers is crucial for practical application.
Key Results
- State-of-the-art performance: The ITL regularizer achieved comparable or better prediction accuracy than baselines (ββ, SGL, CGES) across multiple datasets.
- Significant reduction in FLOPs and parameters: ITL demonstrated substantial reductions in floating-point operations (FLOPs) and the number of parameters used in the network. For example, on MNIST, FLOPs were reduced by nearly 49% and parameters by 16.01% compared to baselines, while maintaining high accuracy.
- Improved convergence speed: Networks trained with ITL converged faster than those trained with other regularizers on the MNIST dataset.
- Effective filter pruning: Visualization of filters in convolutional layers showed that ITL effectively removed redundant features, resulting in sharper and more efficient filters compared to other methods.
- Parameter 'a' impact: Experiments on the DIGITS dataset showed that the choice of parameter 'a' in Tββ significantly impacts accuracy, FLOPs, and parameter usage, with values around 10β»Β² to 10β»ΒΉ often yielding good results.
Limitations and Future Work
- Limited architecture tested: The current work primarily verifies the effectiveness of ITL on convolutional neural networks.
- Future exploration: The authors plan to test ITL on other neural network architectures. They also aim to explore replacing the ββ norm in group sparsity with other regularizers to further enhance the single regularizer's ability to remove both redundant neurons and connections simultaneously.
Summary updated