Pretrained Language Model in Continual Learning: A Comparative Study

Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, Gholamreza HaffariICLR 2022 · OpenReview

Introduction

In the realm of artificial intelligence, humans possess a remarkable ability for "continual learning"—we can learn a new skill today without forgetting the ones we learned yesterday. Neural networks, however, notoriously suffer from Catastrophic Forgetting (CF). When trained on a sequential stream of new tasks, they tend to overwrite the weights optimized for previous tasks, effectively erasing their prior knowledge.

While the Natural Language Processing (NLP) community has actively sought to mitigate CF, existing research has predominantly focused on the original BERT model. This narrow focus leaves several critical questions unanswered: Do newer architectures like ALBERT or XLNet handle forgetting more gracefully? How do specific architectural choices, such as parameter sharing, impact a model's memory? And crucially, what exactly happens inside the "black box" of these models' hidden layers as they sequentially learn?

This paper provides a comprehensive, "under the hood" comparative study to disentangle the interactions between various Pretrained Language Models (PLMs) and Continual Learning (CL) strategies.

Experimental Setup

To systematically evaluate continual learning performance, the researchers conducted a massive benchmarking study combining five major PLMs (BERT, ALBERT, RoBERTa, GPT-2, and XLNet) with four distinct CL approaches:

  • Vanilla: Simple sequential fine-tuning, serving as the baseline for maximum catastrophic forgetting.
  • Regularization-based (EWC): Penalizes significant updates to weights deemed "important" for previous tasks.
  • Dynamic Architecture (HAT): Employs task-specific "masks" to protect certain neural pathways.
  • Rehearsal-based (ER & DERPP): Maintains a small memory buffer of past examples, periodically re-introducing them during training to reinforce old knowledge.

The evaluation utilized three diverse benchmark datasets to simulate real-world challenges: CLINC150 (evenly distributed intent detection), MAVEN (long-tailed event detection), and WebRED (severely long-tailed relation classification). These were split into sequences of tasks to mimic learning over time.

Furthermore, to peer inside the models, the authors introduced Representativeness Probing. Instead of relying solely on final output accuracy, this technique measures how well individual internal layers can still distinguish between classes using a distance-based classification method.

Key Insights and Discoveries

The Superiority of Rehearsal

Across the board, methods that store and replay a small fraction of old examples (Experience Replay) consistently outperformed strategies that attempt to freeze or regularize weights.

Mean Accuracy by Method

As illustrated above, in challenging continual learning settings, the Vanilla baseline often plummets to roughly 5-15% accuracy. In stark contrast, Experience Replay (ER) and its variants (like DERPP) successfully maintain 60-80% accuracy across the same sequence of tasks.

Architectural Quirks: BERT's Reliability vs. ALBERT's Fragility

Despite being one of the older architectures tested, BERT remains a highly reliable "safe bet" for continual learning, frequently demonstrating superior stability compared to more complex models.

Conversely, ALBERT proved to be highly "fragile." ALBERT utilizes cross-layer parameter sharing to reduce memory footprint. However, the study revealed that in a continual learning context, this design is a liability: forgetting in one layer cascades and ripples through the entire network, leading to severe performance degradation.

The "Penultimate Layer Secret"

One of the most fascinating findings emerged from the representativeness probing. The researchers discovered that even when a model experiences catastrophic forgetting and its final accuracy crashes, its internal layers—specifically layers 7 through 12 in BERT and RoBERTa—often retain highly robust representations of older tasks.

This suggests that the "forgetting" is frequently not a complete erasure of knowledge, but rather a calibration failure localized in the final classification head. The core feature extractors remain surprisingly intact.

The Impact of Data Imbalance

The study also highlighted that continual learning is significantly exacerbated by "long-tailed" or imbalanced data distributions. Models are prone to forgetting rare classes much faster than common ones, making datasets like MAVEN and WebRED particularly challenging.

Overall Evaluation Metrics

Conclusion and Future Directions

This comparative study illuminates the nuanced ways different PLMs handle sequential learning. By revealing that internal layers remain robust while the final classifier fails, the authors open exciting avenues for future research. They propose that future CL strategies should pivot toward dynamic adapter structures—modular, task-specific components that can be swapped in and out to prevent the final layer from becoming a catastrophic bottleneck. Additionally, developing automated methods to detect and freeze these "robust layers" in real-time could drastically optimize continual learning efficiency.

Summary updated

Abstract

Continual learning (CL) is a setting in which a model learns from a stream of incoming data while avoiding to forget previously learned knowledge. Pre-trained language models (PLMs) have been successfully employed in continual learning of different natural language problems. With the rapid development of many continual learning methods and PLMs, understanding and disentangling their interactions become essential for continued improvement of continual learning performance. In this paper, we thoroughly compare the continual learning performance over the combination of 5 PLMs and 4 CL approaches on 3 benchmarks in 2 typical incremental settings. Our extensive experimental analyses reveal interesting performance differences across PLMs and across CL methods. Furthermore, our representativeness probing analyses dissect PLMs’ performance characteristics in a layer-wise and task-wise manner, uncovering the extent to which their inner layers suffer from forgetting, and the effect of different CL approaches on each layer. Finally, our observations and analyses open up a number of important research questions that will inform and guide the design of effective continual learning techniques.
BibTeX
@inproceedings{wu2022pretrained,
  author = {Tongtong Wu and Massimo Caccia and Zhuang Li and Yuan-Fang Li and Guilin Qi and Gholamreza Haffari},
  title = {{Pretrained Language Model in Continual Learning: A Comparative Study}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=figzpGMrdD},
}
Figure