Introduction
In the realm of artificial intelligence, humans possess a remarkable ability for "continual learning"—we can learn a new skill today without forgetting the ones we learned yesterday. Neural networks, however, notoriously suffer from Catastrophic Forgetting (CF). When trained on a sequential stream of new tasks, they tend to overwrite the weights optimized for previous tasks, effectively erasing their prior knowledge.
While the Natural Language Processing (NLP) community has actively sought to mitigate CF, existing research has predominantly focused on the original BERT model. This narrow focus leaves several critical questions unanswered: Do newer architectures like ALBERT or XLNet handle forgetting more gracefully? How do specific architectural choices, such as parameter sharing, impact a model's memory? And crucially, what exactly happens inside the "black box" of these models' hidden layers as they sequentially learn?
This paper provides a comprehensive, "under the hood" comparative study to disentangle the interactions between various Pretrained Language Models (PLMs) and Continual Learning (CL) strategies.
Experimental Setup
To systematically evaluate continual learning performance, the researchers conducted a massive benchmarking study combining five major PLMs (BERT, ALBERT, RoBERTa, GPT-2, and XLNet) with four distinct CL approaches:
- Vanilla: Simple sequential fine-tuning, serving as the baseline for maximum catastrophic forgetting.
- Regularization-based (EWC): Penalizes significant updates to weights deemed "important" for previous tasks.
- Dynamic Architecture (HAT): Employs task-specific "masks" to protect certain neural pathways.
- Rehearsal-based (ER & DERPP): Maintains a small memory buffer of past examples, periodically re-introducing them during training to reinforce old knowledge.
The evaluation utilized three diverse benchmark datasets to simulate real-world challenges: CLINC150 (evenly distributed intent detection), MAVEN (long-tailed event detection), and WebRED (severely long-tailed relation classification). These were split into sequences of tasks to mimic learning over time.
Furthermore, to peer inside the models, the authors introduced Representativeness Probing. Instead of relying solely on final output accuracy, this technique measures how well individual internal layers can still distinguish between classes using a distance-based classification method.
Key Insights and Discoveries
The Superiority of Rehearsal
Across the board, methods that store and replay a small fraction of old examples (Experience Replay) consistently outperformed strategies that attempt to freeze or regularize weights.
As illustrated above, in challenging continual learning settings, the Vanilla baseline often plummets to roughly 5-15% accuracy. In stark contrast, Experience Replay (ER) and its variants (like DERPP) successfully maintain 60-80% accuracy across the same sequence of tasks.
Architectural Quirks: BERT's Reliability vs. ALBERT's Fragility
Despite being one of the older architectures tested, BERT remains a highly reliable "safe bet" for continual learning, frequently demonstrating superior stability compared to more complex models.
Conversely, ALBERT proved to be highly "fragile." ALBERT utilizes cross-layer parameter sharing to reduce memory footprint. However, the study revealed that in a continual learning context, this design is a liability: forgetting in one layer cascades and ripples through the entire network, leading to severe performance degradation.
The "Penultimate Layer Secret"
One of the most fascinating findings emerged from the representativeness probing. The researchers discovered that even when a model experiences catastrophic forgetting and its final accuracy crashes, its internal layers—specifically layers 7 through 12 in BERT and RoBERTa—often retain highly robust representations of older tasks.
This suggests that the "forgetting" is frequently not a complete erasure of knowledge, but rather a calibration failure localized in the final classification head. The core feature extractors remain surprisingly intact.
The Impact of Data Imbalance
The study also highlighted that continual learning is significantly exacerbated by "long-tailed" or imbalanced data distributions. Models are prone to forgetting rare classes much faster than common ones, making datasets like MAVEN and WebRED particularly challenging.
Conclusion and Future Directions
This comparative study illuminates the nuanced ways different PLMs handle sequential learning. By revealing that internal layers remain robust while the final classifier fails, the authors open exciting avenues for future research. They propose that future CL strategies should pivot toward dynamic adapter structures—modular, task-specific components that can be swapped in and out to prevent the final layer from becoming a catastrophic bottleneck. Additionally, developing automated methods to detect and freeze these "robust layers" in real-time could drastically optimize continual learning efficiency.
Summary updated