Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression

Huabin Diao, Yuexing Hao, Shaoyun Xu, Gongyan LiSensors 2021 · DOI

TL;DR

This paper introduces a novel "Layer-Wise Differentiable Compression" (LWDC) algorithm that structurally compresses Convolutional Neural Networks (CNNs) by replacing bulky layers with a selection of lightweight operators. This end-to-end trainable method achieves significant model compression with minimal accuracy loss, outperforming existing techniques by directly learning optimal lightweight architectures.

What Problem Does This Paper Address?

Deep learning models, particularly Convolutional Neural Networks (CNNs), have achieved remarkable success in areas like computer vision and natural language processing. However, this performance often comes at the cost of massive model sizes and high computational demands. This makes deploying these powerful models on resource-constrained devices such as mobile phones, embedded systems, or edge devices extremely challenging. Existing compression methods often rely on pruning individual parameters or filters, which can lead to sparse models that require specialized hardware for acceleration and may not achieve end-to-end compression. There's a clear need for methods that can create genuinely smaller and faster CNNs without sacrificing accuracy, and that can be trained efficiently.

Methods

The core innovation of this paper is the Layer-Wise Differentiable Compression (LWDC) algorithm. Instead of pruning existing components, LWDC reconstructs each layer of a CNN into an "over-parameterized" version with multiple parallel branches, each containing a different lightweight convolutional operator.

Here's how it works:

  • Reconstruction Stage: Each original convolutional layer is replaced by a layer with N parallel branches. Each branch houses a pre-defined lightweight operator (e.g., group convolution, depthwise separable convolution, CReLU). A trainable "mask" (a one-hot vector) is applied to these branches, ensuring only one operator is active per layer during training.
  • Searching Stage: This reconstructed, over-parameterized network is trained end-to-end using gradient descent. A key aspect is the use of a differentiable approximation for the discrete one-hot mask, allowing the network to learn which lightweight operator is best suited for each layer. A regularization term is introduced to penalize model size and FLOPs, guiding the search towards more compressed architectures.
  • Fine-Tuning Stage: Once the searching stage is complete, the inactive branches (those with a mask of 0) are removed, leaving a compact, lightweight CNN. This pruned network is then fine-tuned to recover any potential performance degradation.

The method is "differentiable" because it uses a continuous approximation of the discrete selection process, enabling gradient-based optimization. It's "layer-wise" because it makes decisions about operator selection independently for each layer.

Datasets

The proposed LWDC algorithm was evaluated on several standard datasets:

  • CIFAR-10: A dataset of 60,000 32x32 RGB images across 10 classes.
  • CIFAR-100: Similar to CIFAR-10 but with 100 classes.
  • ImageNet-160-120: A down-sampled variant of ImageNet, containing 151.7K training and 6K testing images across 120 classes.

Key Insights and Ideas

  • Replacing, Not Pruning: The fundamental shift from pruning to replacing bulky operators with lightweight ones is a key differentiator. This allows for structural changes that are harder to achieve with traditional pruning.
  • End-to-End Differentiable Compression: The ability to compress and train the network simultaneously using gradient descent simplifies the process and potentially leads to better optima.
  • Automatic Operator Selection: The method automatically searches for the best lightweight operator for each layer, reducing the need for manual hyperparameter tuning of compression rates per layer.
  • Operator Lightness vs. Layer Depth: The study observed a trend where lighter operators tend to be selected for shallower layers (closer to the input), while heavier operators are preferred in deeper layers. This suggests a principled way of distributing computational complexity based on feature map resolution and information redundancy.
  • The Role of $\lambda$ (Regularization Factor): The regularization factor $\lambda$ plays a crucial role in balancing compression and accuracy. A higher $\lambda$ leads to greater compression but can degrade accuracy, while a lower $\lambda$ prioritizes accuracy. The paper shows that this trade-off can be managed effectively.

Key Results

The LWDC algorithm demonstrated strong performance across various CNN architectures and datasets:

  • Significant Compression: The method achieved substantial compression ratios. For example, compressed ResNet20 and ResNet56 retained only 0.11M and 0.24M parameters, respectively, outperforming original models.
  • Competitive Accuracy: Despite high compression rates, the LWDC method generally achieved lower performance degradation compared to state-of-the-art methods. For instance, on CIFAR-10, LWDC achieved higher compression than other methods while maintaining comparable or better accuracy.
  • ImageNet Performance: On ImageNet, LWDC compressed DenseNet121 by 3.42 times with only a 0.23% accuracy drop. For MobileNetV2, it achieved a compression rate of 1.48 with an accuracy improvement.
  • Operator Effectiveness: The study highlighted that more complex lightweight operators (like Fire, Sep_res, Dil_res) can be more effective than simpler ones, even if they are slightly heavier, suggesting a nuanced approach to operator selection.

Limitations and Future Work

  • Hyperparameter Tuning ($\lambda$): While the method reduces hyperparameters compared to some existing techniques, the regularization factor $\lambda$ still requires careful tuning to balance compression and accuracy. The paper provides insights into how $\lambda$ affects these trade-offs.
  • Operator Set Design: The effectiveness of LWDC relies on the pre-defined set of lightweight operators. Designing optimal operator sets for different tasks or architectures could be an area for future exploration.
  • Extensibility: The authors suggest that the differentiable selection operator is generalizable and could be applied to other network types (RNNs, GANs) and tasks (detection, segmentation), as well as for model quantization and neural architecture search (NAS). This opens up significant avenues for future research.

Summary updated

Abstract

Convolutional neural networks (CNNs) have achieved significant breakthroughs in various domains, such as natural language processing (NLP), and computer vision. However, performance improvement is often accompanied by large model size and computation costs, which make it not suitable for resource-constrained devices. Consequently, there is an urgent need to compress CNNs, so as to reduce model size and computation costs. This paper proposes a layer-wise differentiable compression (LWDC) algorithm for compressing CNNs structurally. A differentiable selection operator OS is embedded in the model to compress and train the model simultaneously by gradient descent in one go. Instead of pruning parameters from redundant operators by contrast to most of the existing methods, our method replaces the original bulky operators with more lightweight ones directly, which only needs to specify the set of lightweight operators and the regularization factor in advance, rather than the compression rate for each layer. The compressed model produced by our method is generic and does not need any special hardware/software support. Experimental results on CIFAR-10, CIFAR-100 and ImageNet have demonstrated the effectiveness of our method. LWDC obtains more significant compression than state-of-the-art methods in most cases, while having lower performance degradation. The impact of lightweight operators and regularization factor on the compression rate and accuracy also is evaluated.
BibTeX
@inproceedings{diao2021implementation,
  author = {Huabin Diao and Yuexing Hao and Shaoyun Xu and Gongyan Li},
  title = {{Implementation of Lightweight Convolutional Neural Networks via Layer-Wise Differentiable Compression}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21103464},
}
Figure