Abstract
Model pruning is essential for deploying large Pre-trained Language Models (PLMs) in resource-constrained environments. However, standard task-specific pruning often leads to "catastrophic forgetting" of general language knowledge. We present ContrAstive Pruning (CAP), a framework that leverages contrastive learning to maintain both task-agnostic and task-specific knowledge during compression. By anchoring the pruned model to its pre-trained, fine-tuned, and intermediate versions, CAP maintains high performance and generalization even when removing up to 97% of the model's parameters.
Introduction
Pre-trained Language Models like BERT have revolutionized NLP, but their massive size makes them expensive to deploy. Model pruning—identifying and removing unimportant weights—is a popular solution to reduce model size. However, most existing methods focus exclusively on the specific downstream task during the pruning process. This "task-specific" focus often causes the model to forget the general language knowledge it gained during initial pre-training. This leads to catastrophic forgetting, where the model becomes brittle and sees a sharp performance drop at high sparsity levels.
The CAP Framework
The authors introduce CAP (ContrAstive Pruning), a framework that uses contrastive learning to "pull" the hidden representations of the pruned model toward three different "teacher" sources:
- PrC (Pre-trained Contrastive): Forces the pruned model to mimic the original, unpruned pre-trained model to preserve general language knowledge.
- FiC (Fine-tuned Contrastive): Forces the model to mimic a fully fine-tuned dense version of the model to ensure task-specific accuracy.
- SnC (Snapshot Contrastive): Acts as a bridging mechanism using intermediate models from earlier pruning steps as anchors to provide a smoother transition from dense to sparse.
CAP is a general framework that can be applied to both structured pruning (removing entire blocks or heads) and unstructured pruning (removing individual weights).
Key Results and Performance
The effectiveness of CAP was evaluated on standard benchmarks including MNLI, QQP, SST-2, and SQuAD v1.1. The results indicate that CAP significantly outperforms vanilla pruning methods, especially as sparsity increases.
Performance at Extreme Sparsity
At 97% sparsity (meaning only 3% of the encoder parameters remain), CAP-soft retained 99.2% of the original BERT performance on QQP and 96.3% on MNLI. This demonstrates the framework's ability to maintain core functionality even under extreme compression.
Structured vs. Unstructured Pruning
In structured pruning scenarios at 90% sparsity, CAP improved average scores across tasks by 4.3 points compared to standard first-order pruning. The framework provides a consistent performance boost across different pruning paradigms and sparsity ratios.
Superior Generalization
In "probing" experiments, models pruned with CAP performed significantly better when transferred to new tasks compared to models pruned with vanilla methods. For example, a model pruned on QQP using CAP showed up to a 28.75 point improvement on QNLI over the baseline, confirming that maintaining general knowledge acts as a safety net for generalization.
Conclusion
ContrAstive Pruning (CAP) effectively addresses the challenge of catastrophic forgetting in PLM compression. By utilizing contrastive signals from multiple stages of the model's lifecycle, CAP enables the creation of highly sparse models that remain robust and generalizable. Despite the use of multiple teacher models, the process is efficient, resulting in an extra GPU memory overhead of only 2.86%. Future work may explore scaling these contrastive signals to even larger models like GPT-3 or T5.
Summary updated