Continual Training of Language Models for Few-Shot Learning

Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, Bing Liu

This paper, presented at EMNLP 2022, introduces CPT (Continual Post-Training), a framework designed to help large language models (LMs) learn new domains incrementally without losing their edge on previously learned ones.

TL;DR

CPT is the first system designed for the continual unsupervised post-training of language models to enhance few-shot learning across a sequence of domains. By employing parallel "CL-plugins" and a novel hard-masking mechanism, it achieves zero catastrophic forgetting and overcomes a newly identified challenge called the "Catastrophic Butterfly Effect."

What Problem Does This Paper Address?

Large language models like RoBERTa are powerful, but they perform even better when "post-trained" on unlabeled text from a specific domain (e.g., medical papers or restaurant reviews) before being fine-tuned for a specific task. However, training these models is computationally expensive. Ideally, we want to update an existing LM with new knowledge incrementally as new data becomes available.

The authors identify two major hurdles:

  1. Catastrophic Forgetting (CF): The well-known issue where learning new information causes a model to overwrite and forget previously acquired skills.
  2. Catastrophic Butterfly Effect (CBE): A new phenomenon identified by the authors. In continual learning, even if "soft" masks are used to protect old neurons, tiny gradient leaks (the "butterfly's wings") during post-training can lead to vastly different and degraded results (the "tornado") once the model undergoes the final stage of few-shot fine-tuning.

Methods

The CPT architecture builds upon the RoBERTa backbone by inserting two CL-plugins into each transformer layer.

Architecture of CPT showing CL-plugins in parallel with FFN and Attention layers.

Key technical components include:

  • Parallel Adapters: Unlike standard sequential adapters, CPT places plugins in parallel with the Attention and Feed-Forward Network (FFN) layers. The authors found this parallel structure significantly more effective for continual learning.
  • Task Masks (TMs): CPT uses task IDs to learn differentiable masks that identify which neurons in the plugins are important for a specific domain.
  • Hard Masking for CBE: To solve the "Butterfly Effect," the authors replace traditional soft masks with hard binary masks (using a threshold $\theta=0.5$). This ensures that gradients for protected neurons are exactly zero, preventing any "leakage" that could ruin the model's initialization for future fine-tuning.

During post-training, the backbone LM is frozen, and only the plugins are trained using a Masked Language Model (MLM) loss. During the final few-shot fine-tuning, the entire model (LM + domain-specific plugin neurons) becomes trainable.

Datasets

The authors evaluated CPT using four diverse unlabeled domain datasets for post-training and their corresponding supervised datasets for few-shot end-task evaluation:

  • Restaurant: Yelp Reviews $\rightarrow$ SemEval-res (Aspect Sentiment Classification)
  • AI: AI Papers $\rightarrow$ SCIERC (Relation Classification)
  • ACL: ACL Papers $\rightarrow$ ACL-ARC (Citation Intent Classification)
  • News: AGNews $\rightarrow$ AGNews-FT (News Classification)

The few-shot tasks were extremely constrained, using only 32 to 56 training samples per dataset.

Key Insights and Ideas

  • The Butterfly Effect in Fine-Tuning: The most significant insight is that conventional continual learning evaluation (which tests a model directly) is insufficient for LMs that require a subsequent fine-tuning step. Small changes in weights that seem negligible during post-training can lead to massive performance drops after fine-tuning.
  • Isolation is Key: The results suggest that for few-shot learning, strict parameter isolation (via hard masks) is superior to regularization-based methods (like EWC) or distillation-based methods (like KD), which still allow for some weight drift.
  • Robustness to Order: CPT's performance remains stable regardless of the order in which domains are presented, a common weakness in many continual learning systems.

Key Results

CPT consistently outperformed both non-continual baselines and state-of-the-art continual learning methods.

Table 1: End-task macro-F1 (MF1), accuracy and forgetting rate results for all domains after continual post- training of all domains. The results are averages of 5 random seeds (the domain training order is as they appear in the first row). Due to space limits, the results for different domain orders and the standard deviations are reported in Appendix E and Appendix F, respectively). Non-CL baselines has no forgetting.
  • Zero Forgetting: CPT achieved a forgetting rate of 0.00, whereas other adapted CL methods like HAT (2.43) and BCL (1.47) still suffered from performance decay.
  • Superior Performance: CPT achieved an average Macro-F1 (MF1) of 46.41, significantly higher than vanilla RoBERTa (43.72) and even the "RoBERTa-ONE" baseline (44.78), which trains a separate model for every single task.
  • Ablation Success: As shown in the ablation study, removing the hard-masking fix for the butterfly effect caused the MF1 to drop from 46.41 to 44.17, confirming the necessity of the hard mask.
Ablation study results highlighting the impact of the butterfly effect fix.

Limitations and Future Work

The authors acknowledge two primary limitations:

  1. Architectural Overhead: While the CL-plugins are small relative to the LM, they still add extra parameters to the model.
  2. Requirement of Domain IDs: CPT currently requires a domain ID during both training and testing. In some real-world applications, identifying the specific domain of an incoming task automatically would be more practical.

Future work involves exploring ways to update the underlying LM directly without additional modules and developing methods to handle tasks where domain IDs are not explicitly provided.

Summary updated

Abstract

Recent work on applying large language models (LMs) achieves impressive performance in many NLP applications. Adapting or posttraining an LM using an unlabeled domain corpus can produce even better performance for end-tasks in the domain. This paper proposes the problem of continually extending an LM by incrementally post-train the LM with a sequence of unlabeled domain corpora to expand its knowledge without forgetting its previous skills. The goal is to improve the few-shot end-task learning in these domains. The resulting system is called CPT (Continual PostTraining), which to our knowledge, is the first continual post-training system. Experimental results verify its effectiveness.
BibTeX
@inproceedings{ke2022continual,
  author = {Zixuan Ke and Haowei Lin and Yijia Shao and Hu Xu and Lei Shu and Bing Liu},
  title = {{Continual Training of Language Models for Few-Shot Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=k-1RGc6kBG},
}
Figure