From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression

Runxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Huang, Songfang Huang, Fei Huang

Abstract

Pre-trained Language Models (PLMs) have achieved great success in various Natural Language Processing (NLP) tasks under the pre-training and fine-tuning paradigm. With large quantities of parameters, PLMs are computation-intensive and resource-hungry. Hence, model pruning has been introduced to compress large-scale PLMs. However, most prior approaches only consider task-specific knowledge towards downstream tasks, but ignore the essential task-agnostic knowledge during pruning, which may cause catastrophic forgetting problem and lead to poor generalization ability. To maintain both task-agnostic and task-specific knowledge in our pruned model, we propose ContrAstive Pruning (CAP) under the paradigm of pre-training and fine-tuning. It is designed as a general framework, compatible with both structured and unstructured pruning. Unified in contrastive learning, CAP enables the pruned model to learn from the pre-trained model for task-agnostic knowledge, and fine-tuned model for task-specific knowledge. Besides, to better retain the performance of the pruned model, the snapshots (i.e., the intermediate models at each pruning iteration) also serve as effective supervisions for pruning. Our extensive experiments show that adopting CAP consistently yields significant improvements, especially in extremely high sparsity scenarios. With only 3% model parameters reserved (i.e., 97% sparsity), CAP successfully achieves 99.2% and 96.3% of the original BERT performance in QQP and MNLI tasks. In addition, our probing experiments demonstrate that the model pruned by CAP tends to achieve better generalization ability.

Introduction

Large-scale Pre-trained Language Models (PLMs) like BERT are powerful but resource-intensive, requiring significant memory and computation for deployment. Pruning—the removal of redundant weights—is a standard solution. However, traditional pruning methods often focus exclusively on downstream task performance, causing the model to "forget" the general language knowledge acquired during pre-training. This "catastrophic forgetting" leads to poor generalization and sharp performance drops at high sparsity levels. We propose ContrAstive Pruning (CAP) to bridge this gap, ensuring sparse models remain as adaptable as their dense counterparts.

Methodology: The CAP Framework

CAP is a general framework that can be integrated into existing structured or unstructured pruning algorithms. It leverages contrastive learning to supervise the pruning process by aligning the internal representations of the pruned model with high-quality "teacher" models.

The CAP framework utilizes three contrastive modules (PrC, SnC, FiC) to transfer knowledge from pre-trained, fine-tuned, and snapshot models to the pruned model.

The framework employs three distinct contrastive modules:

  1. PrC (Pre-trained Contrastive): Encourages the pruned model to mimic the original pre-trained model, preserving general language "common sense."
  2. FiC (Fine-tuned Contrastive): Encourages the model to mimic the dense, fine-tuned version, preserving task-specific expertise.
  3. SnC (Snapshot Contrastive): Uses intermediate model snapshots from earlier pruning steps as teachers to provide a smoother transition from dense to sparse.

To address memory overhead, sentence representations are pre-encoded and stored in CPU memory, adding only approximately 2.86% to the total memory footprint.

Experimental Evaluation

We evaluated CAP on standard NLP benchmarks, including MNLI, QQP, SST-2, and SQuAD v1.1. The experiments compared CAP against standard pruning and knowledge distillation methods like TinyBERT and DistillBERT across various sparsity ratios.

Key Results and Discussion

CAP consistently outperformed baseline methods across all tasks, particularly in extreme sparsity scenarios where vanilla pruning methods typically fail.

Performance comparison between pruning with and without CAP across different sparsity levels and pruning types.

Performance at Extreme Sparsity

At 97% sparsity (only 3% parameters remaining), CAP achieved 99.2% of the original BERT performance on QQP and 96.3% on MNLI. As shown in the results, CAP provides consistent gains of 1.0 to 4.3 points over standard movement pruning across different sparsity levels.

Generalization and the "Sparsity Wall"

While standard pruning methods often hit a "wall" at 90% sparsity where performance collapses, CAP pushes this boundary significantly. Probing experiments demonstrate that CAP-pruned models maintain superior generalization ability, scoring up to 28.75 points higher in transfer learning tasks compared to standard methods. This suggests that preserving task-agnostic knowledge strengthens the underlying model architecture.

Conclusion

ContrAstive Pruning (CAP) effectively mitigates catastrophic forgetting in model compression. While highly successful for sentence-level tasks, future work will explore enhancing token-level performance (e.g., for SQuAD) and scaling the framework to even larger architectures like GPT-3 or T5.

Summary updated

Abstract

Pre-trained Language Models (PLMs) have achieved great success in various Natural Language Processing (NLP) tasks under the pre-training and fine-tuning paradigm. With large quantities of parameters, PLMs are computation-intensive and resource-hungry. Hence, model pruning has been introduced to compress large-scale PLMs. However, most prior approaches only consider task-specific knowledge towards downstream tasks, but ignore the essential task-agnostic knowledge during pruning, which may cause catastrophic forgetting problem and lead to poor generalization ability. To maintain both task-agnostic and task-specific knowledge in our pruned model, we propose ContrAstive Pruning (CAP) under the paradigm of pre-training and fine-tuning. It is designed as a general framework, compatible with both structured and unstructured pruning. Unified in contrastive learning, CAP enables the pruned model to learn from the pre-trained model for task-agnostic knowledge, and fine-tuned model for task-specific knowledge. Besides, to better retain the performance of the pruned model, the snapshots (i.e., the intermediate models at each pruning iteration) also serve as effective supervisions for pruning. Our extensive experiments show that adopting CAP consistently yields significant improvements, especially in extremely high sparsity scenarios. With only 3% model parameters reserved (i.e., 97% sparsity), CAP successfully achieves 99.2% and 96.3% of the original BERT performance in QQP and MNLI tasks. In addition, our probing experiments demonstrate that the model pruned by CAP tends to achieve better generalization ability.
BibTeX
@inproceedings{xu2022from,
  author = {Runxin Xu and Fuli Luo and Chengyu Wang and Baobao Chang and Jun Huang and Songfang Huang and Fei Huang},
  title = {{From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21408},
}
Figure