TL;DR
This paper introduces PRACTISE, a novel method for accelerating deep neural networks using only a small subset of training data. It argues that dropping entire network blocks is more effective than traditional filter pruning for this "few-shot" compression scenario, achieving better latency-accuracy trade-offs and proposing a new "recoverability" metric to guide block selection.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are computationally expensive, hindering their deployment on resource-constrained devices. Network compression techniques like pruning aim to reduce this cost. However, these methods typically require access to the entire training dataset for finetuning to recover lost accuracy. This is often infeasible due to data privacy concerns, limited storage, or tight deadlines. The paper tackles the challenge of accelerating DNNs when only a "tiny" (scarce) amount of training data is available, a scenario increasingly relevant in practice.
Methods
The core of PRACTISE lies in two key innovations:
- Block Dropping: The paper argues that dropping entire blocks (e.g., residual blocks in ResNet) is a more effective strategy than filter-level pruning for few-shot compression. This approach offers a higher acceleration ratio, meaning it can achieve significant speedups with fewer FLOPs, making it easier to recover accuracy with limited data.
- Recoverability Metric: To intelligently select which blocks to drop, the authors introduce a new concept called "recoverability." This metric quantifies how easily a pruned network can regain its accuracy after finetuning with a small dataset. It's computed by training only small "adaptor" layers inserted around the dropped block, which is computationally efficient and surprisingly effective at predicting the finetuned accuracy.
- PRACTISE Algorithm: This algorithm combines block dropping with the recoverability metric and a latency consideration. It calculates a "pruning score" for each block based on its recoverability and its contribution to latency. Blocks with the lowest scores are then dropped, and the remaining network is finetuned using a feature-mimicking loss function.
Datasets
- ImageNet-1k: Used for training and evaluating the compressed models (ResNet-34, ResNet-50, MobileNetV2).
- Place365, CUB: Used for evaluating the generalization ability of PRACTISE on out-of-domain datasets.
- Synthesized images from DFKD methods: Used to test PRACTISE in a data-free scenario.
Key Insights and Ideas
- Latency vs. FLOPs Tradeoff: The paper emphasizes that for few-shot compression, the latency-accuracy tradeoff is more critical than the FLOPs-accuracy tradeoff, as real-world performance on devices matters most.
- Block Dropping Superiority: Dropping entire blocks is shown to be a fundamentally superior approach for few-shot compression compared to filter pruning, offering better acceleration and accuracy recovery.
- Recoverability as a Predictive Metric: The proposed "recoverability" metric is a novel and effective way to predict the finetuned accuracy of a pruned network, outperforming existing criteria that don't account for finetuning.
- Adaptor Training Efficiency: Training only small adaptor layers around dropped blocks is sufficient to estimate recoverability, making the process computationally efficient for the few-shot setting.
- Robustness and Generalization: PRACTISE demonstrates strong performance even with synthesized data or out-of-domain datasets, highlighting its versatility.
Key Results
- Significant Accuracy Improvement: For a 22.1% latency reduction on ImageNet-1k, PRACTISE achieved an average of 7.0% higher Top-1 accuracy compared to previous state-of-the-art methods.
- Superior Latency-Accuracy Tradeoff: Figure 1 and various tables show PRACTISE consistently outperforming other methods across different latency reduction levels.
- Efficiency: The method is efficient, with recoverability computation taking minutes and total training time around 1.5 hours for ResNet-34.
- Generalization: PRACTISE works well on out-of-domain datasets and even in data-free scenarios.
- Block Dropping Baseline: Even a simple block dropping baseline (BP (block)) without sophisticated criteria outperformed existing methods in some cases.
Limitations and Future Work
- Focus on Recognition: The current work is primarily focused on image recognition tasks.
- Future Directions: The authors suggest extending PRACTISE to other model architectures (e.g., Transformers) and other vision tasks like object detection and segmentation. They also propose exploring recoverability computation for other compression schemes and applying PRACTISE for fine-tuning large pre-trained models on downstream tasks.
Summary updated