From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression

Runxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Huang, Songfang Huang, Fei Huang

Abstract

Model pruning is essential for deploying large Pre-trained Language Models (PLMs) in resource-constrained environments. However, standard task-specific pruning often leads to "catastrophic forgetting" of general language knowledge. We present ContrAstive Pruning (CAP), a framework that leverages contrastive learning to maintain both task-agnostic and task-specific knowledge during compression. By anchoring the pruned model to its pre-trained, fine-tuned, and intermediate versions, CAP maintains high performance and generalization even when removing up to 97% of the model's parameters.

Introduction

Pre-trained Language Models like BERT have revolutionized NLP, but their massive size makes them expensive to deploy. Model pruning—identifying and removing unimportant weights—is a popular solution to reduce model size. However, most existing methods focus exclusively on the specific downstream task during the pruning process. This "task-specific" focus often causes the model to forget the general language knowledge it gained during initial pre-training. This leads to catastrophic forgetting, where the model becomes brittle and sees a sharp performance drop at high sparsity levels.

The CAP Framework

The authors introduce CAP (ContrAstive Pruning), a framework that uses contrastive learning to "pull" the hidden representations of the pruned model toward three different "teacher" sources:

  1. PrC (Pre-trained Contrastive): Forces the pruned model to mimic the original, unpruned pre-trained model to preserve general language knowledge.
  2. FiC (Fine-tuned Contrastive): Forces the model to mimic a fully fine-tuned dense version of the model to ensure task-specific accuracy.
  3. SnC (Snapshot Contrastive): Acts as a bridging mechanism using intermediate models from earlier pruning steps as anchors to provide a smoother transition from dense to sparse.
The CAP framework anchors the pruned model to pre-trained, fine-tuned, and snapshot versions.

CAP is a general framework that can be applied to both structured pruning (removing entire blocks or heads) and unstructured pruning (removing individual weights).

Key Results and Performance

The effectiveness of CAP was evaluated on standard benchmarks including MNLI, QQP, SST-2, and SQuAD v1.1. The results indicate that CAP significantly outperforms vanilla pruning methods, especially as sparsity increases.

Performance at Extreme Sparsity

At 97% sparsity (meaning only 3% of the encoder parameters remain), CAP-soft retained 99.2% of the original BERT performance on QQP and 96.3% on MNLI. This demonstrates the framework's ability to maintain core functionality even under extreme compression.

Structured vs. Unstructured Pruning

In structured pruning scenarios at 90% sparsity, CAP improved average scores across tasks by 4.3 points compared to standard first-order pruning. The framework provides a consistent performance boost across different pruning paradigms and sparsity ratios.

CAP performance gains across structured and unstructured pruning at various sparsity levels.

Superior Generalization

In "probing" experiments, models pruned with CAP performed significantly better when transferred to new tasks compared to models pruned with vanilla methods. For example, a model pruned on QQP using CAP showed up to a 28.75 point improvement on QNLI over the baseline, confirming that maintaining general knowledge acts as a safety net for generalization.

Conclusion

ContrAstive Pruning (CAP) effectively addresses the challenge of catastrophic forgetting in PLM compression. By utilizing contrastive signals from multiple stages of the model's lifecycle, CAP enables the creation of highly sparse models that remain robust and generalizable. Despite the use of multiple teacher models, the process is efficient, resulting in an extra GPU memory overhead of only 2.86%. Future work may explore scaling these contrastive signals to even larger models like GPT-3 or T5.

Summary updated

Abstract

Pre-trained Language Models (PLMs) have achieved great success in various Natural Language Processing (NLP) tasks under the pre-training and fine-tuning paradigm. With large quantities of parameters, PLMs are computation-intensive and resource-hungry. Hence, model pruning has been introduced to compress large-scale PLMs. However, most prior approaches only consider task-specific knowledge towards downstream tasks, but ignore the essential task-agnostic knowledge during pruning, which may cause catastrophic forgetting problem and lead to poor generalization ability. To maintain both task-agnostic and task-specific knowledge in our pruned model, we propose ContrAstive Pruning (CAP) under the paradigm of pre-training and fine-tuning. It is designed as a general framework, compatible with both structured and unstructured pruning. Unified in contrastive learn- ing, CAP enables the pruned model to learn from the pre-trained model for task-agnostic knowledge, and fine-tuned model for task-specific knowledge. Besides, to better retain the performance of the pruned model, the snapshots (i.e., the intermediate models at each pruning iteration) also serve as effective supervisions for pruning. Our extensive experiments show that adopting CAP consistently yields significant improvements, especially in extremely high sparsity scenarios. With only 3% model parameters reserved (i.e., 97% sparsity), CAP successfully achieves 99.2% and 96.3% of the original BERT performance in QQP and MNLI tasks. In addition, our probing experiments demonstrate that the model pruned by CAP tends to achieve better generalization ability.
BibTeX
@inproceedings{xu2022from,
  author = {Runxin Xu and Fuli Luo and Chengyu Wang and Baobao Chang and Jun Huang and Songfang Huang and Fei Huang},
  title = {{From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21408},
}
Figure