Voting from Nearest Tasks: Meta-Vote Pruning of Pretrained Models for Downstream Tasks

Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang

TL;DR

This paper introduces "Meta-Vote Pruning (MVP)," a novel approach to efficiently prune large pre-trained models for new downstream tasks. MVP leverages the knowledge from previously pruned models of similar tasks to initialize a sub-network, significantly reducing pruning iterations and computational cost while maintaining high accuracy.

What Problem Does This Paper Address?

The widespread adoption of massive pre-trained models (like ResNets and Vision Transformers) has led to a new challenge: how to efficiently adapt these models to a multitude of downstream tasks. Traditional model pruning methods, which aim to reduce model size by removing redundant parameters, are often applied from scratch for each new task. This is computationally expensive, time-consuming, and environmentally taxing. The authors propose to address this by reusing the "pruning knowledge" gained from similar tasks, effectively creating a meta-pruning strategy. They investigate whether pruned models of similar tasks share common structural components (nodes, filters, or heads) and use this insight to build a more efficient pruning pipeline.

Methods

The core of the paper is the Meta-Vote Pruning (MVP) method. MVP operates in two main stages:

  1. Meta-Pruning Initialization:
    • Nearest Task Identification: Given a new target task, MVP first identifies a set of "nearest" tasks from a pre-existing "model zoo" of pruned models. This similarity is determined using metrics like Log Expected Empirical Prediction (LEEP) and WordNet similarity.
  2. Majority Voting: For each layer of the pre-trained model, MVP aggregates the pruned components (filters/nodes/heads) from the identified nearest tasks. A component is more likely to be selected for the new task's sub-network if it was preserved in a larger number of similar tasks' pruned models. This is achieved through a softmax-based voting mechanism.
  3. Parameter Initialization: The parameters of the selected sub-network components are initialized by averaging the corresponding parameters from the nearest tasks' pruned models.
  4. Fine-tuning: The initialized sub-network is then fine-tuned for a small number of iterations on the target task's data. This limited fine-tuning is crucial for keeping the overall computational cost low.

The paper also introduces an empirical study to understand the overlap of pruned models across similar tasks. They employ two pruning techniques for this study: Iterative Filter Pruning (IFP) for CNNs and Automatic Head&Node Pruning (AHNP) for ViTs.

Datasets

The authors created a dataset of pruned models for hundreds of tasks. They used the following pre-trained models and datasets for their empirical study and experiments:

  • CNNs:
    • ResNet-18 pre-trained on CIFAR-100.
  • ResNet-50 pre-trained on ImageNet.
  • Vision Transformers (ViTs):
    • A small version of DeiT pre-trained on ImageNet.

For evaluating MVP on unseen datasets, they used:

  • Caltech-256
  • CUB200-2011
  • Oxford Flowers-102
  • Oxford-IIIT Pets

Key Insights and Ideas

  • Pruning Knowledge Transfer: Similar tasks indeed tend to share more preserved components (filters, nodes, heads) in their pruned models, especially in deeper layers that capture more abstract features. This suggests that pruning decisions are not entirely task-specific and can be generalized.
  • Meta-Pruning Efficiency: By initializing a sub-network from the pruned models of similar tasks, MVP significantly reduces the number of required fine-tuning iterations for a new task, leading to substantial savings in computation and time.
  • LEEP vs. WordNet Similarity: The LEEP score, which considers the learned representations of pruned models, appears to be a more effective metric for identifying similar tasks than semantic similarity metrics like WordNet, especially for deeper layers.
  • Generalization Power: MVP demonstrates strong generalization capabilities, performing well even on unseen datasets and tasks that were not part of the original model zoo.
  • The "Lottery Ticket" Analogy: MVP can be seen as finding a "winning ticket" (a good sub-network initialization) for a new task by leveraging the collective wisdom of winning tickets from similar tasks, rather than searching from scratch.

Key Results

  • Accuracy: MVP consistently achieves comparable or better accuracy than conventional pruning methods (like IFP and AHNP) that use significantly more pruning iterations. For example, MVP achieved 88.98% accuracy on ResNet-18 for CIFAR-100 with 100 iterations, while IFP achieved 87.99% with 1000 iterations.
  • Efficiency: MVP drastically reduces the number of pruning iterations. While conventional methods might require hundreds or thousands of iterations, MVP achieves strong results with around 100 meta-pruning iterations followed by a few fine-tuning steps. This translates to significantly lower FLOPs (Floating Point Operations).
  • Comparison to Baselines: MVP outperforms other meta-learning and pruning methods like IHT-based Reptile, MEST, and DLTH on CNNs, and UVC and PoWER on ViTs, especially under limited pruning iterations.
  • Unseen Datasets: On fine-grained datasets like Caltech-256 and CUB200-2011, MVP's performance was comparable to IFP using 10x more iterations, highlighting its effectiveness in challenging scenarios.
  • Sub-task Sizes: MVP is applicable to tasks of varying sizes, maintaining comparable or better performance than SoTA methods while using much less computation.

Limitations and Future Work

  • Model Zoo Creation Cost: The initial cost of creating the model zoo by pruning models for hundreds of tasks is a one-time overhead. However, the authors argue this is an offline cost and is amortized over many future tasks.
  • Number of Neighbors: While MVP benefits from multiple neighbors, the performance gains plateau after around 3 neighbors, suggesting that a small set of well-chosen similar tasks is sufficient.
  • Future Work: The authors suggest that their meta-pruning approach can be extended to a lifelong learning scenario where new pruned models can be continuously added to the model zoo, further improving future meta-pruning efforts. They also highlight the potential for reducing the carbon footprint of large-scale model pruning.

Summary updated

Abstract

As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task? How to reuse the pruning results of previous tasks to accelerate the pruning for a new task? To address these challenges, we create a small model for a new task from the pruned models of similar tasks. We show that a few fine-tuning steps on this model suffice to produce a promising pruned-model for the new task. We study this ``meta-pruning'' from nearest tasks on two major classes of pre-trained models, convolutional neural network (CNN) and vision transformer (ViT), under a limited budget of pruning iterations. Our study begins by investigating the overlap of pruned models for similar tasks and how the overlap changes over different layers and blocks. Inspired by these discoveries, we develop a simple but effective ``Meta-Vote Pruning (MVP)'' method that significantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models of its nearest tasks. In experiments, we demonstrate MVP's advantages in accuracy, efficiency, and generalization through extensive empirical studies and comparisons with popular pruning methods over several datasets.
BibTeX
@inproceedings{zhao2023voting,
  author = {Haiyan Zhao and Tianyi Zhou and Guodong Long and Jing Jiang and Chengqi Zhang},
  title = {{Voting from Nearest Tasks: Meta-Vote Pruning of Pretrained Models for Downstream Tasks}},
  booktitle = {ECML/PKDD},
  year = {2023},
  doi = {10.1007/978-3-031-43415-0\_4},
}
Figure