LdsConv: Learned Depthwise Separable Convolutions by Group Pruning

Wenxiang Lin, Yan Ding, Hua-Liang Wei, Xinglin Pan, Yutong ZhangSensors 2020 · OpenReview

TL;DR

This paper introduces LdsConv, a novel convolutional filter that integrates learned pruning into depthwise separable convolutions. By intelligently removing less influential filters during training, LdsConv achieves significant reductions in computational cost (FLOPs) and parameter count while simultaneously improving accuracy across various CNN architectures.

What Problem Does This Paper Address?

Standard convolutional filters in deep neural networks are known for their redundancy. They often capture overlapping features, leading to a waste of computational resources and increased model size. While depthwise separable convolutions offer a more efficient alternative by decoupling spatial filtering from channel combination, simply replacing standard convolutions with them can sometimes lead to performance degradation. The authors aim to address this by developing a "smart" convolutional unit that can be seamlessly integrated into existing architectures, offering the benefits of efficiency without sacrificing accuracy, and even improving it.

Methods

The core innovation is the Learned Depthwise Separable Convolution (LdsConv). It's designed as a plug-and-play replacement for standard convolutional layers. The key idea is to integrate a weight pruning technique directly into the filter design and training process. This is achieved through a two-stage training framework:

  1. Picking Stage (Group Pruning): This stage focuses on identifying and removing less influential filters. It uses a group pruning strategy where filters are grouped, and within each group, filters with low L1-norm weights are progressively pruned. A novel balance loss function is introduced to prevent over-pruning of features that are already sparsely represented. This stage is repeated multiple times, controlled by a stage factor (s).
  2. Combining Stage: After the picking stages, an additional pointwise convolution is added. This convolution combines the outputs of the pruned depthwise filters and reconstructs the standard depthwise separable convolution structure. The initial weights of this pointwise convolution are carefully set to leverage the learned filter relationships from the picking stage.

The LdsConv can replace standard convolutions or depthwise convolutions. For standard convolutions, it's typically integrated into a bottleneck block, and for depthwise separable convolutions, it replaces the depthwise part.

Datasets

  • CIFAR-100: Used for ablation studies to fine-tune hyperparameters like pruning factor and stage factor, and to evaluate the effect of the balance loss and group cardinality.
  • ImageNet: Used for extensive experiments to validate the performance of LdsConv on state-of-the-art CNNs (ResNet, DenseNet, MobileNet, SE-ResNet) and compare it with existing compression methods.

Key Insights and Ideas

  • Pruning as a Design Principle: Instead of post-hoc model compression, integrating pruning directly into the filter design and training process allows the network to learn which features are truly valuable, leading to more efficient and accurate representations.
  • Two-Stage Training for Efficiency and Accuracy: The separation of "picking" (pruning) and "combining" (reconstruction) stages simplifies the learning task for each phase, enabling a better balance between computational reduction and accuracy preservation.
  • Intelligent Filter Selection: The L1-norm based pruning criterion, combined with the balance loss, ensures that the most informative filters are retained, preventing the loss of crucial information.
  • Plug-and-Play Nature: The ability to directly replace existing convolutional layers without architectural modifications makes LdsConv highly practical for adoption.
  • Beyond Standard Depthwise Separable Convolutions: LdsConv enhances depthwise separable convolutions by making the selection of filters and their combinations learned and adaptive, rather than fixed.

Key Results

  • Significant FLOPs Reduction: For ResNet50 on ImageNet, LdsConv reduced FLOPs by 40.9% compared to the baseline.
  • Improved Accuracy: LdsConv consistently improved accuracy across various models and datasets. For example, on ImageNet, Lds-MobileNet achieved 2.3% higher accuracy than MobileNet.
  • Parameter Reduction: LdsConv also led to a reduction in the number of parameters. For ResNet50 on ImageNet, parameters were reduced from 25.6M to 16.8M.
  • Competitive Performance: LdsConv outperformed other compression methods like ThiNet, NISP, and FPGM on ResNet50 in terms of accuracy while achieving comparable or better FLOPs reduction.
  • Effective Visualization: Grad-CAM visualizations showed that LdsConv-integrated networks better focus on target regions, indicating improved feature exploitation.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores the effects of pruning factor (k) and stage factor (s), finding the optimal values might require careful tuning for different architectures and datasets. The paper suggests k=2 and s=4 as good general choices.
  • Group Cardinality (NO): The choice of group cardinality (NO) can impact performance, with an "over-grouping" potentially leading to a shrink in expression ability.
  • Future Work: The authors suggest exploring LdsConv in more complex architectures and for other vision tasks. They also mention that their source code will be made available, facilitating further research. The extreme state of LdsConv (Lds-ResNet50-extreme) shows potential for even more aggressive compression, hinting at future directions for highly efficient models.

Summary updated

Abstract

Standard convolutional filters usually capture unnecessary overlap of features resulting in a waste of computational cost. In this paper, we aim to solve this problem by proposing a novel Learned Depthwise Separable Convolution (LdsConv) operation that is smart but has a strong capacity for learning. It integrates the pruning technique into the design of convolutional filters, formulated as a generic convolutional unit that can be used as a direct replacement of convolutions without any adjustments of the architecture. To show the effectiveness of the proposed method, experiments are carried out using the state-of-the-art convolutional neural networks (CNNs), including ResNet, DenseNet, SE-ResNet and MobileNet, respectively. The results show that by simply replacing the original convolution with LdsConv in these CNNs, it can achieve a significantly improved accuracy while reducing computational cost. For the case of ResNet50, the FLOPs can be reduced by 40.9%, meanwhile the accuracy on the associated ImageNet increases.
BibTeX
@inproceedings{lin2020ldsconv,
  author = {Wenxiang Lin and Yan Ding and Hua-Liang Wei and Xinglin Pan and Yutong Zhang},
  title = {{LdsConv: Learned Depthwise Separable Convolutions by Group Pruning}},
  booktitle = {Sensors},
  year = {2020},
  url = {https://openreview.net/forum?id=BFIxr1IBCj},
}
Figure