This paper, presented at EMNLP 2022, introduces CPT (Continual Post-Training), a framework designed to help large language models (LMs) learn new domains incrementally without losing their edge on previously learned ones.
TL;DR
CPT is the first system designed for the continual unsupervised post-training of language models to enhance few-shot learning across a sequence of domains. By employing parallel "CL-plugins" and a novel hard-masking mechanism, it achieves zero catastrophic forgetting and overcomes a newly identified challenge called the "Catastrophic Butterfly Effect."
What Problem Does This Paper Address?
Large language models like RoBERTa are powerful, but they perform even better when "post-trained" on unlabeled text from a specific domain (e.g., medical papers or restaurant reviews) before being fine-tuned for a specific task. However, training these models is computationally expensive. Ideally, we want to update an existing LM with new knowledge incrementally as new data becomes available.
The authors identify two major hurdles:
- Catastrophic Forgetting (CF): The well-known issue where learning new information causes a model to overwrite and forget previously acquired skills.
- Catastrophic Butterfly Effect (CBE): A new phenomenon identified by the authors. In continual learning, even if "soft" masks are used to protect old neurons, tiny gradient leaks (the "butterfly's wings") during post-training can lead to vastly different and degraded results (the "tornado") once the model undergoes the final stage of few-shot fine-tuning.
Methods
The CPT architecture builds upon the RoBERTa backbone by inserting two CL-plugins into each transformer layer.
Key technical components include:
- Parallel Adapters: Unlike standard sequential adapters, CPT places plugins in parallel with the Attention and Feed-Forward Network (FFN) layers. The authors found this parallel structure significantly more effective for continual learning.
- Task Masks (TMs): CPT uses task IDs to learn differentiable masks that identify which neurons in the plugins are important for a specific domain.
- Hard Masking for CBE: To solve the "Butterfly Effect," the authors replace traditional soft masks with hard binary masks (using a threshold $\theta=0.5$). This ensures that gradients for protected neurons are exactly zero, preventing any "leakage" that could ruin the model's initialization for future fine-tuning.
During post-training, the backbone LM is frozen, and only the plugins are trained using a Masked Language Model (MLM) loss. During the final few-shot fine-tuning, the entire model (LM + domain-specific plugin neurons) becomes trainable.
Datasets
The authors evaluated CPT using four diverse unlabeled domain datasets for post-training and their corresponding supervised datasets for few-shot end-task evaluation:
- Restaurant: Yelp Reviews $\rightarrow$ SemEval-res (Aspect Sentiment Classification)
- AI: AI Papers $\rightarrow$ SCIERC (Relation Classification)
- ACL: ACL Papers $\rightarrow$ ACL-ARC (Citation Intent Classification)
- News: AGNews $\rightarrow$ AGNews-FT (News Classification)
The few-shot tasks were extremely constrained, using only 32 to 56 training samples per dataset.
Key Insights and Ideas
- The Butterfly Effect in Fine-Tuning: The most significant insight is that conventional continual learning evaluation (which tests a model directly) is insufficient for LMs that require a subsequent fine-tuning step. Small changes in weights that seem negligible during post-training can lead to massive performance drops after fine-tuning.
- Isolation is Key: The results suggest that for few-shot learning, strict parameter isolation (via hard masks) is superior to regularization-based methods (like EWC) or distillation-based methods (like KD), which still allow for some weight drift.
- Robustness to Order: CPT's performance remains stable regardless of the order in which domains are presented, a common weakness in many continual learning systems.
Key Results
CPT consistently outperformed both non-continual baselines and state-of-the-art continual learning methods.
- Zero Forgetting: CPT achieved a forgetting rate of 0.00, whereas other adapted CL methods like HAT (2.43) and BCL (1.47) still suffered from performance decay.
- Superior Performance: CPT achieved an average Macro-F1 (MF1) of 46.41, significantly higher than vanilla RoBERTa (43.72) and even the "RoBERTa-ONE" baseline (44.78), which trains a separate model for every single task.
- Ablation Success: As shown in the ablation study, removing the hard-masking fix for the butterfly effect caused the MF1 to drop from 46.41 to 44.17, confirming the necessity of the hard mask.
Limitations and Future Work
The authors acknowledge two primary limitations:
- Architectural Overhead: While the CL-plugins are small relative to the LM, they still add extra parameters to the model.
- Requirement of Domain IDs: CPT currently requires a domain ID during both training and testing. In some real-world applications, identifying the specific domain of an incoming task automatically would be more practical.
Future work involves exploring ways to update the underlying LM directly without additional modules and developing methods to handle tasks where domain IDs are not explicitly provided.
Summary updated