TL;DR
This paper introduces Dynamic-Coded Filter Fusion (DCFF), a novel method for creating smaller, more efficient Convolutional Neural Networks (CNNs) without relying on pre-trained models or complex regularization. DCFF dynamically assesses filter importance based on their inter-similarity and fuses them, leading to significant reductions in computational cost and parameters while maintaining high classification accuracy.
What Problem Does This Paper Address?
Deploying powerful CNNs on resource-constrained devices like mobile phones is a major challenge due to their high computational demands and large memory footprints. Existing methods for making CNNs more compact, primarily through filter pruning, often fall into two categories: either they rely on estimating filter importance from a pre-trained, computationally expensive model, or they introduce complex, hyperparameter-sensitive regularization terms into the training loss. Both approaches have drawbacks: the former is dependent on a large model, and the latter can be difficult to tune and generalize. This paper aims to bridge this gap by offering a filter pruning method that is computationally economical, regularization-free, and doesn't require a pre-trained model.
Methods
The core of DCFF lies in its novel approach to filter importance evaluation and fusion:
- Dynamic-Coded Importance: Instead of using static criteria (like L1-norm or feature map rank) on a pre-trained model, DCFF dynamically estimates filter importance during training. It achieves this by:
- Inter-Similarity Distribution Proxy: Each filter is assigned a "proxy" distribution that captures its similarity to all other filters in the same layer. This proxy is influenced by a temperature parameter.
- Kullback-Leibler (KL) Divergence: The importance of a filter is then measured by how much its proxy distribution differs from others, using KL divergence. This captures how unique or representative a filter is within the layer.
- Training-Adaptive Temperature: The temperature parameter is crucial. It starts low, allowing for dynamic changes in filter importance early in training, and gradually increases towards infinity. As it approaches infinity, the proxy distribution becomes a "one-hot" vector, stabilizing the importance scores and the network training.
- Filter Fusion: Unlike methods that simply discard low-importance filters, DCFF proposes "filter fusion." This means the new, compact filters are formed by taking a weighted average of the original filters, with the weights determined by the dynamic importance scores. This approach aims to retain more information from the original network, even from filters that might have had lower individual scores.
- Training from Scratch: A key advantage is that DCFF trains the compact CNN from scratch, eliminating the need for a computationally intensive pre-trained model and subsequent fine-tuning.
Datasets
The paper evaluates DCFF on several standard image classification benchmarks:
- CIFAR-10: Used for training and evaluating compact versions of VGGNet-16, GoogLeNet, and ResNet-56/110.
- ILSVRC-2012: Used for training and evaluating compact versions of ResNet-50 and MobileNet-V2.
Key Insights and Ideas
- Filter Importance is Dynamic: The paper strongly argues that filter importance is not static and can change significantly during training. Relying on importance scores from a pre-trained model or even fixed scores during training can be misleading, as demonstrated by experiments where high-score filters from pre-training lose their importance after fine-tuning.
- Inter-Similarity Matters: Measuring a filter's importance based on its relationship and distinctiveness from other filters in the same layer (using KL divergence on similarity proxies) is more effective than just looking at its intrinsic properties (like L1-norm).
- Fusion Over Pruning: Simply discarding filters can lead to performance degradation. Fusing filters (weighted averaging based on importance) allows for a more graceful reduction in model complexity while preserving crucial information.
- Adaptive Temperature is Key: The training-adaptive temperature parameter is critical for balancing early-stage exploration of filter importance with late-stage stabilization, preventing training instability.
Key Results
DCFF demonstrates strong performance across various architectures and datasets:
- VGGNet-16 on CIFAR-10: Achieved 93.47% top-1 accuracy with a significant reduction of 76.8% in FLOPs and 92.8% in parameters.
- GoogLeNet on CIFAR-10: Outperformed the state-of-the-art HRank method by maintaining higher accuracy (94.92% vs. 94.53%) while reducing more FLOPs (70.1%) and parameters (66.3%).
- ResNet-56/110 on CIFAR-10: Showed significant reductions in FLOPs and parameters, with ResNet-110 accuracy increasing by 0.30%.
- ResNet-50 on ILSVRC-2012: Achieved 75.18% top-1 accuracy with 45.3% FLOPs reduction and 40.7% parameter reduction, outperforming methods like ABCPruner and CURL.
- Training Efficiency: The method trains compact models from scratch, significantly reducing overall processing time compared to methods requiring pre-training and fine-tuning.
Limitations and Future Work
- Per-Layer Pruning Ratio Determination: While DCFF provides a method for determining importance within a layer, the paper notes that determining the optimal number of filters to keep per layer (the pruning ratio) is still manually determined. They suggest that their method can be complemented by automated structure search or global ranking techniques for determining these ratios, which could further boost performance.
- Distance Measurement: While Euclidean distance was found to perform best, the paper explored other distance metrics, implying that the choice of distance metric could influence results.
- Future Work: The authors suggest that combining DCFF with structure searching methods for per-layer pruning ratios is a promising direction for further improvement.
Summary updated