Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models

Ajay Jaiswal, Shiwei Liu, Tianlong Chen, Ying Ding, Zhangyang Wang

Introduction

In the current landscape of artificial intelligence, the trend toward larger models has yielded unprecedented performance but at a significant computational "tax." Massive architectures like Transformers are prohibitively expensive to deploy and fine-tune. To mitigate these costs, researchers employ pruning—the process of removing redundant connections to create efficient, sparse subnetworks.

The gold standard for identifying these subnetworks is the Lottery Ticket Hypothesis (LTH). LTH posits that dense networks contain sparse "winning tickets" that, when trained in isolation, can match the performance of the original dense model. Traditionally, finding these tickets requires Iterative Magnitude Pruning (IMP), a cycle of training, pruning, and retraining. While effective, IMP is computationally expensive, often requiring multiple full training passes, making it impractical for modern billion-parameter models.

The Instant Soup Pruning (ISP) Framework

This paper introduces Instant Soup Pruning (ISP), a novel method designed to find high-quality sparse subnetworks within the budget of a single standard training pass. ISP replaces the expensive iterative cycles of IMP with a strategic "denoising" approach based on model ensembles.

1. Sparsity-Dependent Training

ISP is built on the observation that pruning difficulty is non-linear. At low sparsity levels (e.g., removing 10–30% of weights), the task is relatively trivial. ISP allocates minimal computational resources to these early stages and gradually increases training intensity as the network becomes thinner and the pruning task becomes more complex.

2. Mask Denoising (The "Soup")

Rather than relying on a single, potentially noisy mask generated from one training trajectory, ISP performs several "weak" and extremely short training iterations using different data subsets and protocols. While individual masks from these runs are noisy, ISP takes the union of these masks. This "mask soup" effectively cancels out stochastic noise, leaving behind a robust, high-quality subnetwork.

3. Single-Pass Efficiency

By optimizing the training schedule and using the denoising ensemble, ISP achieves state-of-the-art pruning performance without exceeding the computational cost of a single standard training pass. The authors also extend these insights to Instant Model Soup (IMS), a method for improving the performance of standard dense models with negligible extra overhead.

Experimental Methodology

The researchers evaluated ISP across a diverse range of benchmarks and architectures, focusing on two primary heavyweights:

  • Vision: Evaluated on MNIST, SVHN, Oxford-IIIT Pets, GTSRB, CIFAR-10, and CIFAR-100 using CLIP (ViT-B32). This represents one of the first comprehensive studies on pruning CLIP architectures.
  • Language: Evaluated on the GLUE Benchmark (MNLI, QNLI, QQP, SST-2, STS-B, WNLI, MRPC, RTE, and CoLA) using BERT-BASE.

Key Insights and Results

The "Trivial Sparsity" Realization

The study reveals that expensive pruning methods are often unnecessary at low sparsity levels. Simple one-shot pruning performs comparably to complex methods until approximately 70% of the model is removed. The "heavy lifting" of ISP’s denoising is specifically targeted at high-sparsity regimes where traditional cheap methods typically fail.

Denoising as a Performance Multiplier

The most significant finding is that an ensemble of "weak" masks can outperform a single "strong" mask. This approach allows the model to bypass the need for reaching a stable local minimum before a valid mask can be identified. The authors found that using 4 to 5 denoising iterations provides the optimal balance between accuracy gains and computational efficiency.

Performance Benchmarks

  • CLIP on CIFAR-100: ISP outperformed the computationally intensive LTH-Rewind method by approximately 5.38% while maintaining a significantly faster execution time.
  • BERT on GLUE: ISP dominated standard Lottery Ticket results on 8 out of 9 GLUE tasks. Notably, on the QNLI task, ISP improved upon the baseline by more than 1%.
  • Early-Bird Tickets: While previous research into "early-bird" tickets suggested they could be found early in training, they often lagged in final performance. ISP demonstrates that with proper denoising, these early tickets can actually surpass the performance of expensive iterative methods.

Limitations and Future Directions

While ISP provides a powerful empirical solution for efficient pruning, the authors acknowledge that the theoretical mechanics of why the mask union operation denoises so effectively require further study. Additionally, while the results on CLIP and BERT-BASE are promising, future research is needed to determine if these benefits scale to ultra-large Generative Pre-trained Transformers (LLMs) with hundreds of billions of parameters. Finally, exploring more sophisticated aggregation methods beyond a simple union could potentially unlock even higher levels of sparsity.

Summary updated

Abstract

Large pre-trained transformers have been receiving explosive attention in the past few years, due to their wide adaptability for numerous downstream applications via fine-tuning, but their exponentially increasing parameter counts are becoming a primary hurdle to even just fine-tune them without industry-standard hardware. Recently, Lottery Ticket Hypothesis (LTH) and its variants, have been exploited to prune these large pre-trained models generating subnetworks that can achieve similar performance as their dense counterparts, but LTH pragmatism is enormously inhibited by repetitive full training and pruning routine of iterative magnitude pruning (IMP) which worsens with increasing model size. Motivated by the recent observations of model soups, which suggest that fine-tuned weights of multiple models can be merged to a better minima, we propose Instant Soup Pruning (ISP) to generate lottery ticket quality subnetworks, using a fraction of the original IMP cost by replacing the expensive intermediate pruning stages of IMP with computationally efficient weak mask generation and aggregation routine. More specifically, during the mask generation stage, ISP takes a small handful of iterations using varying training protocols and data subsets to generate many weak and noisy subnetworks, and superpose them to average out the noise creating a high-quality denoised subnetwork. Our extensive experiments and ablation on two popular large-scale pre-trained models: CLIP (unexplored in pruning till date) and BERT across multiple benchmark vision and language datasets validate the effectiveness of ISP compared to several state-of-the-art pruning methods. Codes are available at: \url{https://github.com/VITA-Group/instant_soup}
BibTeX
@inproceedings{jaiswal2023instant,
  author = {Ajay Jaiswal and Shiwei Liu and Tianlong Chen and Ying Ding and Zhangyang Wang},
  title = {{Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=CuWORvLAnp},
}
Figure