TL;DR
This paper introduces a novel "pruning from scratch" framework for convolutional neural networks (CNNs) that bypasses the need for expensive pre-training. By employing a Shared Pruning Module (SPM) for efficient channel alignment in residual blocks and a nuclear norm-based regularization to preserve network representability, they achieve state-of-the-art compression with competitive accuracy on ImageNet and CIFAR10.
What Problem Does This Paper Address?
Deep neural networks, particularly CNNs, have achieved remarkable success but often come with high computational costs, making them challenging to deploy on resource-constrained devices. Network pruning is a popular technique to reduce this complexity by removing redundant components. However, most existing structured pruning methods rely on a costly three-step pipeline: pre-training a large network, pruning it, and then fine-tuning. This paper addresses the inefficiency of pre-training and the challenge of effectively pruning networks from scratch, especially when dealing with the structural complexities of modern architectures like residual blocks. Specifically, it tackles the "channel alignment problem" in residual blocks, where pruning outer layers without careful consideration can lead to performance degradation or increased model size during reconfiguration.
Methods
The core of this work lies in two main innovations:
- Shared Pruning Module (SPM): To address the channel alignment problem in residual blocks, the authors propose the SPM. This module ensures that the same pruning mask is applied to the outer layers of a residual block, guaranteeing that the number of channels in the feature maps remains consistent after reconfiguration. This allows for efficient pruning and lossless reconfiguration without increasing the network's size.
- Nuclear Norm-Based Regularization: To preserve the "representability" (the network's ability to capture and express complex patterns) of the pruned network, a novel regularization term, $L_{nuc}$, is introduced. This term leverages the nuclear norm of the pruned layer's weights, which is a more flexible measure of representability than traditional rank. This is combined with a MACs-based regularization ($L_{mac}$) to guide the pruning towards the target computational budget. The overall regularization loss is $L_{reg} = \alpha_{nuc}L_{nuc} + \alpha_{mac}L_{mac}$.
The framework operates by pruning from scratch (starting with randomly initialized weights) and integrates both regularization terms into the training process.
Datasets
The paper evaluates its method on two widely used datasets:
- CIFAR10: A dataset with 10 classes, used for training and evaluating the pruned ResNet56 model.
- ImageNet (ILSVRC 2012): A large-scale dataset with 1,000 classes, used for training and evaluating the pruned ResNet50 model.
Key Insights and Ideas
- Pruning from scratch is viable and efficient: The paper demonstrates that it's possible to achieve competitive results without the computationally intensive pre-training step, significantly improving the overall efficiency of the pruning process.
- SPM elegantly solves channel alignment: The Shared Pruning Module is a clever solution to a common architectural challenge in residual networks, enabling efficient reconfiguration without performance loss or model bloat.
- Nuclear norm preserves representability: Using the nuclear norm as a regularization term is effective in maintaining the network's capacity to learn complex features, which is crucial for preventing performance degradation during pruning.
- Balancing MACs and representability is key: The interplay between $L_{mac}$ and $L_{nuc}$ is critical. $L_{mac}$ drives compression, while $L_{nuc}$ ensures the pruned network retains its expressive power, leading to better overall performance.
Key Results
- ImageNet: The proposed method achieves a top-1 accuracy of 75.3% on ImageNet using ResNet50, with only 41% of the original model's MACs, and importantly, without requiring pre-training.
- CIFAR10: On CIFAR10, the pruned ResNet56 model achieves 93.5% accuracy with 51% of its original size, outperforming prior works that utilize pre-trained networks.
- Efficiency: The SPM-based reconfiguration process preserves MACs after pruning, unlike some prior methods (e.g., PaS) where MACs can increase post-reconfiguration. This leads to more practical computational complexity.
Limitations and Future Work
- Limitations: While the paper demonstrates strong results, the authors acknowledge that the nuclear norm regularization's impact on performance is evaluated across various $\alpha_{nuc}$ values, suggesting that finding the optimal balance might require tuning. The paper also notes that while latency reduction is significant, it's not as drastic as MACs reduction due to memory access constraints.
- Future Work: The authors suggest investigating the generalizability of their framework by applying it to other learning tasks beyond image classification.
Summary updated