One-Shot Pruning for Fast-adapting Pre-trained Models on Devices

Haiyan Zhao, Guodong LongarXiv 2023 · DOI · arXiv

Introduction

Large-scale pre-trained models, such as Vision Transformers (ViT) and CLIP, have revolutionized computer vision. However, their massive parameter counts pose significant challenges for deployment on edge devices with limited memory and computational power. While model pruning is a standard technique to reduce model size, traditional iterative pruning methods are computationally expensive and slow, making them impractical for devices that need to adapt quickly to new tasks or user environments.

This paper introduces Scalable Mask Selection Pruning (SMSP), a framework designed to enable one-shot pruning and fast adaptation. By leveraging a pool of previously pruned models from similar tasks, SMSP identifies the critical components of a network for a new task without the need for exhaustive, iterative searching.

The SMSP Framework

The core philosophy of SMSP is that tasks sharing semantic or structural similarities also share important neural pathways. The framework operates in four primary stages:

  1. Model Pool Construction: A library of "mask scores" is pre-computed from a diverse set of tasks using Automatic Mask Pruning (AMP). These scores rank the importance of filters in CNNs or heads and nodes in ViTs.
  2. Task Similarity Estimation: When a new target task is encountered, its similarity to tasks in the pool is calculated using the Log Expected Empirical Prediction (LEEP) metric. LEEP provides a computationally efficient way to estimate transferability.
  3. One-Shot Mask Generation: SMSP selects the top $M$ most similar tasks from the pool and aggregates their importance scores to generate a custom pruning mask for the new task.
  4. Scalable Pruning and Adaptation: The model is pruned to the desired size in a single step and then fine-tuned for a minimal number of iterations to achieve peak performance.
Figure 1: Overview of the SMSP workflow, showing how a pool of existing pruned models is used to generate a mask for a new target task in one shot.

Key Insights: The Similarity Overlap

A fundamental discovery of this research is the strong correlation between task similarity and architectural overlap. As shown in the experimental analysis, tasks categorized into high-similarity groups (Group 1) exhibit a significantly higher overlap ratio of important heads and nodes compared to low-similarity groups (Group 3). This empirical evidence validates the intuition that knowledge from previous pruning efforts can be directly transferred to accelerate the adaptation of new, related tasks.

Figure 3: Overlap ratios of heads and nodes for Vision Transformers (ViT). Higher similarity groups consistently show greater overlap in critical architectural components.

Experimental Results

The researchers evaluated SMSP across benchmarks including CIFAR-100, ImageNet, and Caltech-256. Key findings include:

  • Efficiency: On ResNet-18, SMSP reached 88.55% accuracy in just 100 iterations, matching the performance of traditional methods that required 1,000 iterations.
  • Computational Savings: For Vision Transformers, SMSP reduced the pruning process's computational cost from 81.50 TFLOPs to a mere 3.25 TFLOPs—a 25x reduction.
  • Hardware Awareness: Because SMSP generates a ranked list of importance scores rather than a fixed architecture, it is inherently scalable. The same importance scores can be used to prune a model by 50% for a smartphone or 90% for a low-power IoT device.
  • Generalization: The method generalized well to the unseen Caltech-256 dataset, achieving high accuracy with 5x fewer training iterations than from-scratch pruning.

Conclusion

Scalable Mask Selection Pruning (SMSP) represents a significant step toward making large pre-trained models viable for real-time edge adaptation. By shifting the pruning paradigm from iterative search to similarity-based selection, SMSP drastically reduces the time and energy required to deploy specialized, efficient models on device. Future work may explore extending this "knowledge-sharing" approach to natural language processing and other complex neural architectures.

Summary updated

Abstract

Large-scale pre-trained models have been remarkably successful in resolving downstream tasks. Nonetheless, deploying these models on low-capability devices still requires an effective approach, such as model pruning. However, pruning the model from scratch can pose a practical challenge given the limited resources of each downstream task or device. To tackle this issue, we present a scalable one-shot pruning method that leverages pruned knowledge of similar tasks to extract a sub-network from the pre-trained model for a new task. Specifically, we create a score mask using the pruned models of similar tasks to identify task-specific filters/nodes in the pre-trained model for the new task. Based on this mask, we conduct a single round of pruning to extract a suitably-sized sub-network that can quickly adapt to the new task with only a few training iterations. Our experimental analysis demonstrates the effectiveness of the proposed method on the convolutional neural networks (CNNs) and vision transformers (ViT) with various datasets. The proposed method consistently outperforms popular pruning baseline methods in terms of accuracy and efficiency when dealing with diverse downstream tasks with different memory constraints.
BibTeX
@misc{zhao2023one,
  author = {Haiyan Zhao and Guodong Long},
  title = {{One-Shot Pruning for Fast-adapting Pre-trained Models on Devices}},
  year = {2023},
  eprint = {2307.04365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.04365},
}
Figure