Differentiable Channel Pruning Search.

Yu Zhao, Chung-Kuei LeearXiv 2020 · arXiv

Abstract

In this paper, we propose the differentiable channel pruning search (DCPS) of convolutional neural networks. Unlike traditional channel pruning algorithms which require users to manually set prune ratio for each convolutional layer, DCPS searches for the optimal combination of prune ratios automatically. Inspired by differentiable architecture search (DARTS), we draw lessons from continuous relaxation and leverage gradient information to balance metrics and performance. However, directly applying the DARTS scheme causes channel mismatching problems and huge memory consumption. Therefore, we introduce a novel weight sharing technique which elegantly eliminates the shape mismatching problem with negligible additional resource overhead. We test the proposed algorithm on image classification tasks, achieving state-of-the-art pruning results on CIFAR-10, CIFAR-100, and ImageNet. DCPS is further utilized for semantic segmentation on PASCAL VOC 2012 to demonstrate that task-specific channel pruning achieves better performance than transferring slim models and to prove the memory efficiency of DCPS in memory-intensive tasks. Results validate the effectiveness and wide applicability of DCPS.

Introduction

To deploy deep learning models on mobile and edge devices, channel pruning—the removal of redundant filters to reduce computational cost (FLOPs)—is essential. However, traditional pruning faces a significant challenge: manually determining the "Goldilocks" prune ratio for every layer in a deep network is a massive, tedious search space. If you prune too little, the model remains slow; prune too much, and accuracy collapses.

While Neural Architecture Search (NAS) can automate this process, standard differentiable methods like DARTS struggle with pruning due to two primary issues:

  1. Shape Mismatch: Element-wise operations (like addition) fail when feature maps have different channel counts (e.g., 32 vs. 64 channels).
  2. Memory Explosion: Standard NAS creates separate paths for every candidate size, which quickly exhausts GPU memory.

Differentiable Channel Pruning Search (DCPS) addresses these by introducing a "weight sharing within filters" technique, enabling efficient, gradient-based searching that avoids massive memory overhead.

Methods

DCPS treats the number of channels in a layer as a searchable parameter optimized via gradient descent. The framework relies on several core innovations:

  • Continuous Relaxation: Using the Gumbel-Softmax trick, discrete choices (e.g., selecting between 16, 32, or 64 channels) are transformed into a differentiable probability distribution. This allows the network to "learn" the optimal architecture during training.
  • Weight Sharing within Filters: Instead of separate kernels for every possible channel count, DCPS uses one large "super-kernel." Smaller candidate kernels are represented as the first $N$ channels of that super-kernel.
  • Feature Weighting: Rather than summing different-sized tensors, the method weights the slices of the output feature map. This bypasses shape mismatch and requires almost no extra memory compared to training a standard unpruned model.
Comparison of different channel pruning search strategies, with (e) representing the proposed weight sharing and feature weighting mechanism.

The optimization follows a two-stage process: a "warming-up" stage to stabilize weights, followed by a "searching" stage where architecture gates are optimized on a validation set.

Key Results and Insights

The authors validated DCPS across image classification (CIFAR, ImageNet), semantic segmentation (PASCAL VOC 2012), and image super-resolution (DIV2K).

Performance on ImageNet

On ResNet-18, DCPS achieved 69.83% Top-1 accuracy with only 0.96 GFLOPs. As shown in the figure below, DCPS (referred to as DCCS in some benchmarks) consistently hits the "Pareto front," providing a superior trade-off between speed and accuracy compared to state-of-the-art methods like TAS, FBS, and DCP.

Performance comparison on ImageNet with ResNet-18. DCPS achieves a superior trade-off between Top-1 accuracy and GFLOPs.

Task-Specific Pruning

A key finding was that models pruned specifically for semantic segmentation perform better than models pruned for classification and then transferred. This proves that redundancy is task-dependent; a network optimized for one task may have different bottlenecks than one optimized for another. For ResNet-50 on CIFAR-100, the method achieved a 53.8% prune ratio with a negligible 0.29% drop in accuracy.

Conclusion

DCPS demonstrates that the structure of a network (the number of channels per layer) is often more critical than specific inherited weight values. By sharing weights within filters, the search process becomes feasible even for memory-heavy tasks like segmentation. Future work may extend this differentiable search to more complex structures, such as channel attention mechanisms or depthwise-separable convolutions, and incorporate direct hardware-latency optimization.

Summary updated

Abstract

In this paper, we propose the differentiable channel pruning search (DCPS) of convolutional neural networks. Unlike traditional channel pruning algorithms which require users to manually set prune ratio for each convolutional layer, DCPS search the optimal combination of prune ratio that automatically. Inspired by the differentiable architecture search (DARTS), we draws lessons from the continuous relaxation and leverages the gradient information to balance the metrics and performance. However, directly applying the DARTS scheme will cause channel mismatching problem and huge memory consumption. Therefore, we introduce a novel weight sharing technique which can elegantly eliminate the shape mismatching problem with negligible additional resource. We test the proposed algorithm on image classification task and it achieves the state-of-the-art pruning results for image classification on CIFAR-10, CIFAR-100 and ImageNet. DCPS is further utilized for semantic segmentation on PASCAL VOC 2012 for two purposes. The first is to demonstrate that task-specific channel pruning achieves better performance against transferring slim models, and the second is to prove the memory efficiency of DCPS as the task demand more memory budget than classification. Results of the experiments validate the effectiveness and wide applicability of DCPS.
BibTeX
@misc{zhao2020differentiable,
  author = {Yu Zhao and Chung-Kuei Lee},
  title = {{Differentiable Channel Pruning Search.}},
  year = {2020},
  eprint = {2010.14714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.14714},
}
Figure