TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models

Ziqing Yang, Yiming Cui, Zhigang ChenACL 2022

TL;DR

This paper introduces TextPruner, an open-source toolkit for efficiently pruning pre-trained language models (PLMs) to reduce their size and computational cost. It offers flexible, optimization-free methods for both vocabulary and transformer pruning, making model compression accessible even to users with limited training experience.

What Problem Does This Paper Address?

Pre-trained language models (PLMs) have revolutionized NLP, but their massive size and computational demands make them impractical for many real-world applications. Deploying these models on resource-constrained devices like mobile phones or edge devices is a significant challenge. This paper addresses the need for efficient and user-friendly tools to compress PLMs without requiring extensive retraining or specialized hardware.

Methods

TextPruner offers a suite of pruning techniques designed for ease of use and speed. Its core methods include:

  • Vocabulary Pruning: This technique removes tokens from the model's vocabulary that appear infrequently in a given corpus. This is particularly effective for multilingual models, allowing them to be tailored to specific languages and significantly reducing the embedding matrix size.
  • Transformer Pruning: This method focuses on reducing the size of the transformer architecture itself. It employs structured pruning, removing entire attention heads or intermediate neurons in the feed-forward networks (FFNs). The importance of these components is determined using a sensitivity-based score (how much the loss changes with respect to the component).
  • Optimization-Free Pruning: TextPruner primarily uses optimization-free methods, meaning it doesn't require fine-tuning or retraining the model after pruning. This makes the process much faster and more convenient.
  • Self-Supervised Pruning: A novel contribution is the self-supervised pruning method. Instead of relying on labeled data and the training loss, it uses the KL divergence between the original model's output and the pruned model's output to estimate neuron importance. This allows pruning without any labeled data.
  • Pipeline Pruning: This mode combines both vocabulary and transformer pruning for maximum model compression.

The toolkit provides both a Python API and a command-line interface (CLI) for accessibility.

Datasets

  • XNLI (Cross-lingual Natural Language Inference): Used as the primary dataset for text classification experiments. It's a multilingual dataset, allowing for evaluation of in-language and zero-shot performance.
  • SQuAD (Stanford Question Answering Dataset): Used for evaluating the model's performance on the Machine Reading Comprehension (MRC) task.
  • CoNLL-2003: Used for evaluating the model's performance on the Named Entity Recognition (NER) task.

Key Insights and Ideas

  • Vocabulary pruning is surprisingly effective for multilingual models: Even when pruning tokens from both English and Chinese training sets, the model size can be reduced by ~60% with only a slight performance drop. This suggests that core semantic understanding neurons are shared across languages.
  • Self-supervised pruning is a viable alternative: The self-supervised pruning method, which doesn't require labeled data, achieves comparable or even better results than supervised pruning. This significantly broadens the applicability of pruning.
  • Uneven pruning (UHF) outperforms homogenous pruning: Allowing transformer layers to have different numbers of attention heads and FFN sizes (Uneven Heads and FFN neurons - UHF) leads to better performance than forcing all layers to have the same structure. This indicates that some layers are inherently more important than others.
  • Pruning can be done quickly: For a base-sized model, pruning can be completed in just a few minutes on a modern GPU.
  • Minimal data is needed for effective pruning: The experiments show that using as little as 70% of the development set for importance score calculation can yield performance nearly comparable to using the full set.

Key Results

  • Vocabulary Pruning: For XLM-R on XNLI, pruning vocabulary on both English and Chinese training sets reduced model size by ~60% while maintaining high accuracy (e.g., 85.8% on English test, 74.5% on Chinese test).
  • Transformer Pruning:
    • On XNLI, reducing the model to (6, 1536) (half attention heads, half FFN size) resulted in a significant reduction in transformer size and speedup (Table 2), with performance drops of only a few percentage points.
  • On SQuAD, increasing the number of iterations (n_iters) consistently improved F1 scores, with n_iters=8 achieving good performance.
  • On CoNLL-2003, UHF models with reduced structures like (6, 1536) UHF achieved F1 scores of 87.3%, a substantial improvement over lower iteration counts.
  • Self-supervised pruning achieved comparable or better results than supervised pruning across various configurations.

Limitations and Future Work

  • Limitations: The paper focuses on optimization-free, structured pruning. While this is fast and convenient, it might not achieve the absolute best performance compared to methods that involve retraining. The paper also primarily focuses on base-sized models.
  • Future Work: The authors plan to extend TextPruner to support more PLMs, including generative models like T5. They also aim to integrate TextPruner with their knowledge distillation toolkit, TextBrewer, to offer a unified framework for model compression.

Summary updated

BibTeX
@inproceedings{yang2022textpruner,
  author = {Ziqing Yang and Yiming Cui and Zhigang Chen},
  title = {{TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure