Continual Learning of Language Models

Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, Bing LiuarXiv 2023 · OpenReview

Abstract

Domain-Adaptive Pre-training (DAP) is essential for specializing language models (LMs) for specific fields. However, when applied sequentially to multiple domains, models typically suffer from catastrophic forgetting. This paper introduces DAS (Continual DA-pre-training with Soft-masking), a framework designed to incrementally learn from new domains while preserving general language capabilities and previous domain knowledge. DAS utilizes a robustness proxy to identify critical neurons and a soft-masking mechanism to protect them, achieving state-of-the-art performance and demonstrating positive knowledge transfer across domains.

Introduction

Pre-trained language models like RoBERTa are typically static after their initial training on general corpora. To adapt them to specialized fields such as law or medicine, researchers use Domain-Adaptive Pre-training (DAP). The challenge arises in a continual learning setting: as a model learns a sequence of new domains, it often loses its grasp on earlier ones—a phenomenon known as catastrophic forgetting (CF). Existing continual learning methods often focus on specific end-tasks, but DAS addresses the more fundamental problem of updating the underlying language model itself to remain a versatile base for future applications.

Methods

The DAS framework is built upon three innovative pillars designed to balance plasticity (learning new things) and stability (retaining old things):

  1. Soft-Masking Mechanism: Rather than rigidly freezing weights, DAS computes importance scores for attention heads and neurons. During backpropagation, gradients are scaled by $(1 - \text{importance})$, allowing the model to update less-critical units while protecting "knowledge anchors."
  2. Robustness Proxy (Initialization): To protect general knowledge without needing the original massive pre-training data, DAS uses model robustness as a proxy. By feeding the same input through the model twice with different dropout masks, the system identifies neurons whose outputs fluctuate significantly as critical for stability.
  3. Contrastive Knowledge Integration: To prevent redundant learning, a contrastive loss encourages the network to produce representations that complement previously learned knowledge, forcing the model to focus on novel information in the current domain.
Overview of the DAS framework, illustrating initialization for general knowledge importance and the continual learning process for domain training.

Experimental Setup

The study evaluated DAS using a RoBERTa-base model across six diverse unlabeled domain corpora:

  • Reviews: Yelp Restaurant, Amazon Phone, and Amazon Camera.
  • Academic Papers: ACL Papers (NLP), AI Papers, and PubMed (Biomedical).

Performance was measured by fine-tuning the resulting models on specific end-tasks, including Aspect Sentiment Classification (ASC), Citation Intent Classification, and Chemical-Protein Interaction Prediction.

Results and Discussion

DAS significantly outperformed 16 baselines, including established methods like Elastic Weight Consolidation (EWC) and Knowledge Distillation (KD).

Performance comparison of DAS against various Non-CL and CL baselines across multiple domains, showing superior Macro-F1 scores and negative forgetting rates.

Key Findings:

  • Superior Performance: DAS achieved an average Macro-F1 of 77.93, compared to the naive continual training (NCL) score of 76.36. It performed comparably to, and sometimes better than, the "Pool" benchmark (training on all domains simultaneously).
  • Negative Forgetting: DAS demonstrated a forgetting rate of -1.09. This indicates "backward transfer," where learning a new domain actually improved the model's performance on previously learned domains.
  • Importance of General Knowledge: The results suggest that maintaining the original general knowledge base is critical; if the foundation erodes, domain-specific performance suffers regardless of the training method.

Conclusion and Future Work

DAS provides a robust solution for the continual pre-training of language models by intelligently protecting critical neurons through soft-masking. While highly effective, future research could focus on merging the initialization and training steps into a single seamless flow and testing the scalability of the robustness proxy on multi-billion parameter models.

Summary updated

BibTeX
@misc{ke2023continual,
  author = {Zixuan Ke and Yijia Shao and Haowei Lin and Tatsuya Konishi and Gyuhak Kim and Bing Liu},
  title = {{Continual Learning of Language Models}},
  year = {2023},
  url = {https://openreview.net/forum?id=imJN0MUZkbA},
}
Figure