TL;DR
WHC (Weighted Hybrid Criterion) is a new data-independent filter pruning method that combines filter magnitude with inter-filter relationship data. By identifying filters that are both small in magnitude and redundant in function, it can prune over 42% of FLOPs from a ResNet-50 on ImageNet with zero loss in top-5 accuracy.
What Problem Does This Paper Address?
Deep learning models are notoriously resource-hungry, making them difficult to deploy on mobile or edge devices. Filter pruning—removing entire convolutional filters to create "slimmer" networks—is a popular solution. However, choosing which filters to cut is tricky.
Existing "data-independent" criteria (which don't require looking at training data to decide what to prune) usually fall into two camps:
- Norm-based: These assume filters with small weights (low L1/L2 norms) are unimportant. This fails when most filters have similar magnitudes.
- Relationship-based: These look at how similar filters are to one another (e.g., Euclidean distance or cosine similarity), assuming the most "central" or redundant filters can be removed.
The authors point out that both have "blind spots." For example, a relationship-based approach might prune a unique, orthogonal filter just because it has a small norm, or it might fail to distinguish between two filters that are highly correlated but have different magnitudes.
Methods
The authors introduce the Weighted Hybrid Criterion (WHC). Instead of choosing between magnitude and relationship, WHC integrates them into a single score.
The importance score for a filter is calculated by multiplying its own norm by a weighted sum of its "dissimilarity" to every other filter in the layer. Specifically:
- Magnitude: It uses the $L_2$ norm of the filter.
- Dissimilarity: It uses $(1 - |\cos \theta|)$, where $\theta$ is the angle between two filters. This values orthogonality—filters that extract unique, non-overlapping features.
- The "Weighted" Secret Sauce: When calculating how redundant a filter is, WHC weights the dissimilarity by the norm of the other filters. In other words, being different from a "heavyweight" (high-norm) filter is more important than being different from a "lightweight" one.
The pruning process follows a simple Pretrain-Prune-Finetune pipeline. It is "one-shot," meaning all layers are pruned simultaneously at a fixed rate before a final fine-tuning step, avoiding the complexity of iterative pruning or reinforcement learning.
Datasets
The researchers evaluated WHC on two standard computer vision benchmarks:
- CIFAR-10: Using ResNet-20, 32, 56, and 110.
- ImageNet (ILSVRC-2012): Using ResNet-18, 34, 50, and 101.
They compared their results against a wide array of state-of-the-art methods, including SFP, FPGM, and even data-dependent methods like HRank and ABC.
Key Insights and Ideas
- The Robustness of Orthogonality: Relationship-based methods like FPGM often struggle because they might keep filters that are "antiphase" (180 degrees apart), which are actually highly redundant. WHC’s use of absolute cosine similarity ensures that both identical and perfectly opposite filters are recognized as redundant.
- Counterpart Weighting: The "aha moment" of the paper is that not all relationships are equal. If Filter A is similar to Filter B, but Filter B has a tiny norm and barely contributes to the network, Filter A shouldn't be penalized for that similarity. By weighting the dissimilarity by the counterpart's norm, WHC focuses on redundancy relative to the most "active" parts of the network.
- Simplicity Wins: Despite using a simple one-shot pruning mechanism, WHC often outperforms much more complex, data-driven, or AutoML-based pruning strategies.
Key Results
- ImageNet Performance: On ResNet-50, WHC reduced FLOPs by 42.2% with 0.00% loss in top-5 accuracy. For comparison, the popular SFP method lost 8.27% accuracy at a similar pruning rate.
- Efficiency on Deep Networks: On ResNet-110 (CIFAR-10), WHC pruned 52.3% of FLOPs and actually improved accuracy by 0.39%, suggesting the pruning acted as a form of regularization that reduced overfitting.
- State-of-the-Art (SOTA) Comparison: WHC consistently achieved lower accuracy degradation than norm-based (PFEC, SFP) and relationship-based (FPGM, CFP) criteria across almost all tested depths and datasets.
Limitations and Future Work
- Manual Pruning Rates: While WHC is excellent at ranking filters, the user still has to manually set the pruning rate ($r$) for the layers. The authors suggest that combining WHC with automated methods to find optimal per-layer pruning rates could yield even better results.
- Metric Exploration: The ablation study showed that using $L_1$ norms or correlation coefficients instead of $L_2$ and cosine similarity could slightly change results. The authors note that finding the "perfect" pair of norm and similarity metrics for specific architectures remains an open area for exploration.
- One-Shot Limitation: While simple, one-shot pruning might not be as optimal as iterative pruning for extremely high compression ratios (e.g., removing 90%+ of a model).
Summary updated