Continual Training of Language Models for Few-Shot Learning

Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, Bing Liu

Introduction

Large Language Models (LMs), such as RoBERTa, have demonstrated remarkable capabilities across various natural language processing tasks. However, to achieve peak performance in specialized fields like medicine or law, these models often require "post-training" on domain-specific, unlabeled text. In dynamic environments where new domains emerge sequentially, the challenge is to update the model without losing previously acquired knowledge—a phenomenon known as Catastrophic Forgetting (CF).

While numerous Continual Learning (CL) strategies have been developed to mitigate CF, this paper identifies a more insidious obstacle in the context of few-shot learning: the Catastrophic Butterfly Effect (CBE).

The Catastrophic Butterfly Effect

In traditional CL research, models are typically evaluated directly on tasks after training. However, in the paradigm of post-training, the continually trained model serves as an initialization point for subsequent few-shot fine-tuning. The authors discover that even when a model appears to have "remembered" a previous domain (showing minimal performance drop in zero-shot settings), the minute weight updates allowed by standard CL methods act as a different random initialization.

Much like the metaphorical butterfly flapping its wings and causing a hurricane, these "negligible" gradient shifts lead to massive performance collapses once the model is fine-tuned on a small set of labeled data. Consequently, traditional CL methods that allow for even slight weight drift are insufficient for maintaining a stable foundation for few-shot learning.

Methodology: Continual Post-Training (CPT)

To address both CF and CBE, the authors introduce Continual Post-Training (CPT). The framework utilizes a modular architecture based on adapters—small, trainable components integrated into a frozen transformer backbone.

CL-Plugins and Parallel Architecture

CPT employs "CL-plugins," which are task-specific modules added to each transformer layer. Unlike standard sequential adapters, these plugins are inserted in parallel to the Attention and Feed-Forward Network (FFN) layers. This parallel configuration was found to be more effective for the continual post-training objective.

Task Masks and Hard Masking

The core innovation of CPT is its "hard masking" mechanism. For each new domain, the model learns a Task Mask (TM) that identifies specific neurons within the CL-plugins essential for that domain.

  • Hard Thresholding: While many methods use "soft" masks (continuous values between 0 and 1), CPT applies a strict binary threshold (e.g., 0.5). This ensures that gradients are completely blocked from neurons associated with previous domains during the training of a new domain.
  • Weight Preservation: During the fine-tuning phase for a specific task, only the neurons identified by that domain's hard mask are activated. This guarantees that the initialization remains identical, regardless of how many subsequent domains the model has learned, effectively neutralizing the Butterfly Effect.

Experimental Setup

The researchers evaluated CPT across four diverse domains, using large unlabeled corpora for post-training and small labeled sets for few-shot evaluation:

  1. Restaurant: Yelp reviews (post-training); SemEval-res (few-shot classification).
  2. AI: S2ORC AI papers (post-training); SCIERC (relation classification).
  3. ACL: ACL anthology papers (post-training); ACL-ARC (citation intent classification).
  4. AGNews: News articles (post-training); AGNews-FT (news classification).

CPT was compared against 13 baselines, including standard RoBERTa, traditional adapters, and state-of-the-art CL methods such as EWC, HAT, and DER++.

Results and Analysis

The experimental results highlight the superiority of the CPT framework in maintaining stability across sequential learning tasks.

  • Performance: CPT achieved an average Macro-F1 score of 46.41, significantly outperforming the best-performing CL baseline (BCL at 44.45) and the base RoBERTa model (43.72).
  • Zero Forgetting: Most notably, CPT achieved a forgetting rate of 0.00%. While other CL methods experienced performance drops between 1.4% and 4.9% as new domains were added, CPT’s hard masking ensured that knowledge of the first domain remained perfectly intact after learning subsequent domains.
  • Ablation Studies: The importance of the "hard" threshold was confirmed through ablation. When the threshold was removed—allowing for the Butterfly Effect—the Macro-F1 score dropped from 46.41 to 44.17, underscoring that strict parameter isolation is vital for few-shot stability.

Conclusion

CPT provides a robust solution for the continual post-training of language models, specifically addressing the Catastrophic Butterfly Effect that plagues few-shot fine-tuning. By utilizing parallel CL-plugins and hard binary masking, the framework achieves state-of-the-art performance with zero forgetting.

Future work may address current limitations, such as the requirement for a "Domain ID" during inference and the additional parameter overhead of the plugins, potentially by exploring direct weight updates to the underlying language model.

Summary updated

BibTeX
@inproceedings{ke2022continual,
  author = {Zixuan Ke and Haowei Lin and Yijia Shao and Hu Xu and Lei Shu and Bing Liu},
  title = {{Continual Training of Language Models for Few-Shot Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=k-1RGc6kBG},
}
Figure