TL;DR
This paper introduces "Meta-Vote Pruning (MVP)," a novel approach to efficiently prune large pre-trained models for new downstream tasks. MVP leverages the knowledge from previously pruned models of similar tasks to initialize a sub-network, significantly reducing pruning iterations and computational cost while maintaining high accuracy.
What Problem Does This Paper Address?
The widespread adoption of massive pre-trained models (like ResNets and Vision Transformers) has led to a new challenge: how to efficiently adapt these models to a multitude of downstream tasks. Traditional model pruning methods, which aim to reduce model size by removing redundant parameters, are often applied from scratch for each new task. This is computationally expensive, time-consuming, and environmentally taxing. The authors propose to address this by reusing the "pruning knowledge" gained from similar tasks, effectively creating a meta-pruning strategy. They investigate whether pruned models of similar tasks share common structural components (nodes, filters, or heads) and use this insight to build a more efficient pruning pipeline.
Methods
The core of the paper is the Meta-Vote Pruning (MVP) method. MVP operates in two main stages:
- Meta-Pruning Initialization:
- Nearest Task Identification: Given a new target task, MVP first identifies a set of "nearest" tasks from a pre-existing "model zoo" of pruned models. This similarity is determined using metrics like Log Expected Empirical Prediction (LEEP) and WordNet similarity.
- Majority Voting: For each layer of the pre-trained model, MVP aggregates the pruned components (filters/nodes/heads) from the identified nearest tasks. A component is more likely to be selected for the new task's sub-network if it was preserved in a larger number of similar tasks' pruned models. This is achieved through a softmax-based voting mechanism.
- Parameter Initialization: The parameters of the selected sub-network components are initialized by averaging the corresponding parameters from the nearest tasks' pruned models.
- Fine-tuning: The initialized sub-network is then fine-tuned for a small number of iterations on the target task's data. This limited fine-tuning is crucial for keeping the overall computational cost low.
The paper also introduces an empirical study to understand the overlap of pruned models across similar tasks. They employ two pruning techniques for this study: Iterative Filter Pruning (IFP) for CNNs and Automatic Head&Node Pruning (AHNP) for ViTs.
Datasets
The authors created a dataset of pruned models for hundreds of tasks. They used the following pre-trained models and datasets for their empirical study and experiments:
- CNNs:
- ResNet-18 pre-trained on CIFAR-100.
- ResNet-50 pre-trained on ImageNet.
- Vision Transformers (ViTs):
- A small version of DeiT pre-trained on ImageNet.
For evaluating MVP on unseen datasets, they used:
- Caltech-256
- CUB200-2011
- Oxford Flowers-102
- Oxford-IIIT Pets
Key Insights and Ideas
- Pruning Knowledge Transfer: Similar tasks indeed tend to share more preserved components (filters, nodes, heads) in their pruned models, especially in deeper layers that capture more abstract features. This suggests that pruning decisions are not entirely task-specific and can be generalized.
- Meta-Pruning Efficiency: By initializing a sub-network from the pruned models of similar tasks, MVP significantly reduces the number of required fine-tuning iterations for a new task, leading to substantial savings in computation and time.
- LEEP vs. WordNet Similarity: The LEEP score, which considers the learned representations of pruned models, appears to be a more effective metric for identifying similar tasks than semantic similarity metrics like WordNet, especially for deeper layers.
- Generalization Power: MVP demonstrates strong generalization capabilities, performing well even on unseen datasets and tasks that were not part of the original model zoo.
- The "Lottery Ticket" Analogy: MVP can be seen as finding a "winning ticket" (a good sub-network initialization) for a new task by leveraging the collective wisdom of winning tickets from similar tasks, rather than searching from scratch.
Key Results
- Accuracy: MVP consistently achieves comparable or better accuracy than conventional pruning methods (like IFP and AHNP) that use significantly more pruning iterations. For example, MVP achieved 88.98% accuracy on ResNet-18 for CIFAR-100 with 100 iterations, while IFP achieved 87.99% with 1000 iterations.
- Efficiency: MVP drastically reduces the number of pruning iterations. While conventional methods might require hundreds or thousands of iterations, MVP achieves strong results with around 100 meta-pruning iterations followed by a few fine-tuning steps. This translates to significantly lower FLOPs (Floating Point Operations).
- Comparison to Baselines: MVP outperforms other meta-learning and pruning methods like IHT-based Reptile, MEST, and DLTH on CNNs, and UVC and PoWER on ViTs, especially under limited pruning iterations.
- Unseen Datasets: On fine-grained datasets like Caltech-256 and CUB200-2011, MVP's performance was comparable to IFP using 10x more iterations, highlighting its effectiveness in challenging scenarios.
- Sub-task Sizes: MVP is applicable to tasks of varying sizes, maintaining comparable or better performance than SoTA methods while using much less computation.
Limitations and Future Work
- Model Zoo Creation Cost: The initial cost of creating the model zoo by pruning models for hundreds of tasks is a one-time overhead. However, the authors argue this is an offline cost and is amortized over many future tasks.
- Number of Neighbors: While MVP benefits from multiple neighbors, the performance gains plateau after around 3 neighbors, suggesting that a small set of well-chosen similar tasks is sufficient.
- Future Work: The authors suggest that their meta-pruning approach can be extended to a lifelong learning scenario where new pruned models can be continuously added to the model zoo, further improving future meta-pruning efforts. They also highlight the potential for reducing the carbon footprint of large-scale model pruning.
Summary updated