Introduction
Large-scale pre-trained models, such as Vision Transformers (ViT) and CLIP, have revolutionized computer vision. However, their massive parameter counts pose significant challenges for deployment on edge devices with limited memory and computational power. While model pruning is a standard technique to reduce model size, traditional iterative pruning methods are computationally expensive and slow, making them impractical for devices that need to adapt quickly to new tasks or user environments.
This paper introduces Scalable Mask Selection Pruning (SMSP), a framework designed to enable one-shot pruning and fast adaptation. By leveraging a pool of previously pruned models from similar tasks, SMSP identifies the critical components of a network for a new task without the need for exhaustive, iterative searching.
The SMSP Framework
The core philosophy of SMSP is that tasks sharing semantic or structural similarities also share important neural pathways. The framework operates in four primary stages:
- Model Pool Construction: A library of "mask scores" is pre-computed from a diverse set of tasks using Automatic Mask Pruning (AMP). These scores rank the importance of filters in CNNs or heads and nodes in ViTs.
- Task Similarity Estimation: When a new target task is encountered, its similarity to tasks in the pool is calculated using the Log Expected Empirical Prediction (LEEP) metric. LEEP provides a computationally efficient way to estimate transferability.
- One-Shot Mask Generation: SMSP selects the top $M$ most similar tasks from the pool and aggregates their importance scores to generate a custom pruning mask for the new task.
- Scalable Pruning and Adaptation: The model is pruned to the desired size in a single step and then fine-tuned for a minimal number of iterations to achieve peak performance.
Key Insights: The Similarity Overlap
A fundamental discovery of this research is the strong correlation between task similarity and architectural overlap. As shown in the experimental analysis, tasks categorized into high-similarity groups (Group 1) exhibit a significantly higher overlap ratio of important heads and nodes compared to low-similarity groups (Group 3). This empirical evidence validates the intuition that knowledge from previous pruning efforts can be directly transferred to accelerate the adaptation of new, related tasks.
Experimental Results
The researchers evaluated SMSP across benchmarks including CIFAR-100, ImageNet, and Caltech-256. Key findings include:
- Efficiency: On ResNet-18, SMSP reached 88.55% accuracy in just 100 iterations, matching the performance of traditional methods that required 1,000 iterations.
- Computational Savings: For Vision Transformers, SMSP reduced the pruning process's computational cost from 81.50 TFLOPs to a mere 3.25 TFLOPs—a 25x reduction.
- Hardware Awareness: Because SMSP generates a ranked list of importance scores rather than a fixed architecture, it is inherently scalable. The same importance scores can be used to prune a model by 50% for a smartphone or 90% for a low-power IoT device.
- Generalization: The method generalized well to the unseen Caltech-256 dataset, achieving high accuracy with 5x fewer training iterations than from-scratch pruning.
Conclusion
Scalable Mask Selection Pruning (SMSP) represents a significant step toward making large pre-trained models viable for real-time edge adaptation. By shifting the pruning paradigm from iterative search to similarity-based selection, SMSP drastically reduces the time and energy required to deploy specialized, efficient models on device. Future work may explore extending this "knowledge-sharing" approach to natural language processing and other complex neural architectures.
Summary updated