Holistic Filter Pruning for Efficient Deep Neural Networks

Lukas Enderich, Fabian Timm, Wolfram BurgardWACV 2021

TL;DR

This paper introduces Holistic Filter Pruning (HFP), a novel method that integrates filter pruning directly into the training of deep neural networks. HFP automatically determines optimal pruning rates for individual layers to meet user-defined targets for both parameter count and computational cost (multiplications), achieving state-of-the-art performance on CIFAR-10 and ImageNet with significant model compression.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have become incredibly powerful but also notoriously large and computationally expensive. This "over-parameterization" is often necessary to ensure good initial weights during training, but it leads to redundant filters that can be removed to make models smaller and faster. Existing pruning methods often struggle with this, either requiring extensive manual tuning, being inefficient for large networks, or failing to achieve a global, optimal reduction across all layers. Specifically, pruning individual weights (unstructured sparsity) doesn't translate well to hardware efficiency, while pruning entire filters (structured sparsity) is more promising but challenging to optimize globally.

Methods

The core of HFP lies in its integration of a pruning loss directly into the standard DNN training objective. Instead of pruning after training, HFP prunes during training. The key innovation is using the channel-wise scaling factors of Batch Normalization (BN) layers as a proxy for filter importance.

Here's how it works:

  • BN Scaling Factors as Importance Indicators: BN layers have learnable parameters, including scaling factors ($\gamma$) and biases ($\beta$). When these scaling factors are folded into the preceding convolutional layer (a common optimization for inference), they effectively act as a multiplier for each channel's output. HFP leverages this by treating the absolute value of $\gamma$ as an indicator of a filter's importance. Small $|\gamma|$ values suggest the filter's output is being scaled down significantly, making it a candidate for pruning.
  • Holistic Pruning Loss: HFP introduces a pruning loss that penalizes deviations from a user-defined target for both the total number of parameters ($P^*$) and floating-point multiplications ($M^*$). This loss is calculated after each forward pass.
  • Gradient Descent for Global Optimization: By using a straight-through estimator to approximate the gradient of a non-smooth indicator function (which determines if a filter's scaling factor is below a threshold), HFP can backpropagate the pruning loss. This allows gradient descent to find a global solution that distributes the pruning budget across all layers to minimize the combined learning and pruning loss.
  • No Additional Variables: Crucially, HFP doesn't require introducing new learnable parameters for pruning. It solely relies on the existing BN scaling factors.
  • Fine-tuning: After the training phase, filters identified as pruned (those with scaling factors below a threshold) are permanently removed from the network architecture. The network is then fine-tuned for a few epochs to adapt the remaining filters and update BN statistics.

Datasets

The paper evaluates HFP on two standard image classification benchmarks:

  • CIFAR-10: A dataset of 32x32 color images across 10 classes.
  • ImageNet: A large-scale dataset with 1000 classes, used for evaluating performance on larger models.

Key Insights and Ideas

  • BN Scaling Factors are Powerful Pruning Proxies: The paper demonstrates that the learnable scaling factors in Batch Normalization layers are surprisingly effective indicators of filter importance, allowing for structured pruning without needing separate importance metrics.
  • Integrated Pruning is More Effective: By incorporating pruning directly into the training loop, HFP can achieve better results than post-training pruning methods, especially when aiming for aggressive compression.
  • Global Optimization for Balanced Compression: HFP's ability to distribute pruning across all layers to meet both parameter and multiplication targets simultaneously is a significant advantage over layer-wise or independent pruning strategies.
  • Simplicity and Efficiency: The method is easy to implement, requiring only a few lines of code, and doesn't introduce extra learnable parameters, making it computationally efficient.

Key Results

HFP achieves state-of-the-art performance on CIFAR-10 and ImageNet across various architectures (VGG, ResNet).

  • CIFAR-10 (VGG-8): Achieved 82.0% reduction in FLOPs and 90.0% reduction in parameters with a Top-1 accuracy of 94.21%, outperforming existing methods.
  • CIFAR-10 (ResNet-56): Achieved 76.09% FLOPs reduction and 71.58% parameter reduction with 92.31% Top-1 accuracy, demonstrating strong performance even with aggressive pruning.
  • ImageNet (ResNet-50): Achieved up to 78.24% FLOPs reduction with only a 2% drop in accuracy, and up to 68.48% parameter reduction.
  • ImageNet (ResNet-18): Achieved 36.30% FLOPs reduction with only a 0.6% accuracy decrease, showcasing effectiveness even on smaller, less over-parameterized networks.

The paper also shows that HFP can dynamically allocate pruning rates across layers based on whether the target is parameter reduction or FLOPs reduction, demonstrating its flexibility.

Limitations and Future Work

  • Threshold Selection: While the paper uses a fixed threshold ($t=10^{-4}$) for determining pruned filters, this value might require tuning for different architectures or datasets.
  • Shortcut Connections: The paper addresses shortcut connections (common in ResNets) by ensuring the pruning rate calculation accounts for them, but this remains a complex aspect of pruning.
  • Future Work: The authors suggest exploring more sophisticated methods for determining the pruning parameter $\lambda$ and investigating HFP's applicability to other network architectures and tasks. They also highlight the potential for further research into the dynamic allocation of pruning budgets.

Summary updated

BibTeX
@inproceedings{enderich2021holistic,
  author = {Lukas Enderich and Fabian Timm and Wolfram Burgard},
  title = {{Holistic Filter Pruning for Efficient Deep Neural Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure