Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks

Haiyan Zhao, Tianyi Zhou, Guodong Long, Jing Jiang, Chengqi Zhang

Introduction

The deployment of large-scale foundation models, such as ResNet and Vision Transformers (ViT), onto resource-constrained edge devices remains a significant hurdle in machine learning. While network pruning—the process of removing redundant weights to create smaller, more efficient sub-networks—is the standard solution, it comes with a high computational cost. Traditionally, researchers perform iterative pruning from scratch for every new downstream task. This "start-from-zero" approach is not only computationally expensive but also environmentally taxing, ignoring the potential knowledge stored in previously pruned models.

This paper introduces Meta-Vote Pruning (MVP), a novel framework that treats pruning as a meta-learning problem. By leveraging a "model zoo" of previously pruned tasks, MVP initializes sub-networks for new tasks through a majority voting mechanism, achieving state-of-the-art accuracy with a fraction of the computational effort.

The Core Concept: Task Similarity and Mask Overlap

The fundamental hypothesis of MVP is that similar tasks should result in similar pruned architectures. To test this, the authors conducted an empirical study analyzing hundreds of pruned models. They measured the Intersection over Union (IoU) of the "masks" (the binary indicators of which filters or nodes are preserved) across different tasks.

Their findings confirmed a strong correlation: tasks that are semantically or statistically similar (e.g., different subsets of animal species) tend to preserve the same structural components of a pre-trained model. This overlap is particularly pronounced in the deeper layers of the network, which capture high-level features. This observation suggests that the "optimal" architecture for a new task is likely hidden within the aggregate knowledge of similar existing tasks.

The Meta-Vote Pruning (MVP) Algorithm

MVP operates through a streamlined, non-parametric process that avoids the need for training complex meta-networks:

  1. Identifying Nearest Tasks: When presented with a new target task, the system identifies the $N$ most similar tasks from a pre-existing "model zoo." Similarity is determined using metrics like LEEP (Log Expected Empirical Prediction), which measures data-driven perception, or WordNet semantic similarity.
  2. Majority Voting: Each of the $N$ nearest tasks "votes" for the filters it retained during its own pruning process. Filters that receive a higher number of votes are assigned a higher probability of being included in the new sub-network.
  3. Weight Initialization: Rather than resetting weights to the original pre-trained values or using random initialization, MVP initializes the selected filters by averaging the weights from the corresponding filters in the nearest tasks' pruned models.
  4. Efficiency Fine-tuning: Because the architecture and weights are already well-positioned, the resulting sub-network requires only a "light" fine-tuning phase (e.g., 100 iterations) to reach peak performance.

Experimental Results

The researchers evaluated MVP using ResNet-18, ResNet-50, and DeiT architectures across a variety of benchmarks, including CIFAR-100, ImageNet, and fine-grained datasets like Oxford Flowers and CUB200-2011.

Efficiency and Speedup

The most striking result is the efficiency gain. MVP achieves comparable or superior accuracy to standard Iterative Fine-Pruning (IFP) while using only 10% of the iterations (100 iterations for MVP versus 1,000 for IFP). This 10x speedup significantly reduces the energy consumption required for model deployment.

Accuracy Gains

On ImageNet using a ResNet-50 backbone, MVP reached 91.80% accuracy in just 100 iterations. In contrast, the standard IFP baseline reached only 91.16% accuracy, even after a full 1,000 iterations. MVP also consistently outperformed other sparse training and meta-learning baselines, such as MEST and DLTH.

Generalization

The study demonstrated that MVP generalizes well across different data distributions. Even when the model zoo was constructed primarily from ImageNet-based tasks, the voting logic successfully pruned high-performing models for specialized datasets like Oxford-IIIT Pets and Caltech-256.

Key Insights

  • Data-Driven Similarity Matters: The authors found that LEEP (which looks at how a model perceives data) is a more effective metric for finding "nearest tasks" than WordNet (which relies on human-defined semantic categories).
  • Skipping the Search Phase: Most pruning methods spend the majority of their time searching for the correct architecture. MVP effectively skips this phase by utilizing prior knowledge, moving straight to fine-tuning.
  • Simplicity Scales: Unlike parametric meta-learning approaches that require training a separate model to predict masks, MVP’s voting mechanism is simple, transparent, and easy to scale as the model zoo grows.

Limitations and Future Directions

While MVP offers a significant leap in efficiency, it does require a pre-existing library of pruned models (the "model zoo"). While such libraries are becoming more common in industrial settings, the method's effectiveness is tied to the diversity of the zoo. Additionally, the current implementation uses a fixed pruning ratio (e.g., 90% sparsity). Future research could investigate whether the voting mechanism can also be used to dynamically determine the optimal sparsity level for a specific task, further optimizing the trade-off between model size and performance.

Conclusion

Meta-Vote Pruning represents a shift in how we approach model compression. By recognizing that new tasks are rarely entirely unique, MVP allows developers to "stand on the shoulders" of previous pruning efforts. The result is a faster, greener, and more accurate path to deploying powerful AI on the devices we use every day.

Summary updated

Abstract

As a few large-scale pre-trained models become the major choices of various applications, new challenges arise for model pruning, e.g., can we avoid pruning the same model from scratch for every downstream task? How to reuse the pruning results of previous tasks to accelerate the pruning for a new task? To address these challenges, we create a small model for a new task from the pruned models of similar tasks. We show that a few fine-tuning steps on this model suffice to produce a promising pruned-model for the new task. We study this ''meta-pruning'' from nearest tasks on two major classes of pre-trained models, convolutional neural network (CNN) and vision transformer (ViT), under a limited budget of pruning iterations. Our study begins by investigating the overlap of pruned models for similar tasks and how the overlap changes over different layers and blocks. Inspired by these discoveries, we develop a simple but effective ''Meta-Vote Pruning (MVP)'' method that significantly reduces the pruning iterations for a new task by initializing a sub-network from the pruned models of its nearest tasks. In experiments, we demonstrate MVP's advantages in accuracy, efficiency, and generalization through extensive empirical studies and comparisons with popular pruning methods over several datasets.
BibTeX
@inproceedings{zhao2023voting,
  author = {Haiyan Zhao and Tianyi Zhou and Guodong Long and Jing Jiang and Chengqi Zhang},
  title = {{Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for Downstream Tasks}},
  booktitle = {ECML/PKDD},
  year = {2023},
  doi = {10.1007/978-3-031-43415-0\_4},
}
Figure