Abstract
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, Diyi Yang. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
A fundamental challenge in machine learning is catastrophic forgetting, where a model trained on a sequence of tasks abruptly loses the knowledge required for previous tasks. In Natural Language Processing (NLP), this manifests when a classifier trained on news topics loses its ability to perform sentiment analysis after being fine-tuned on Wikipedia categories.
Existing approaches typically rely on either replay-based methods (storing and re-training on old data) or regularization-based methods (constraining parameter changes). However, standard regularization treats all information as equally important. This paper argues that linguistic features should be treated differently: task-generic information (e.g., syntax) should remain stable, while task-specific information (e.g., sentiment-bearing keywords) requires more flexibility.
Methodology: Information Disentanglement Based Regularization (IDBR)
The authors propose IDBR, a framework that explicitly separates text representations into task-generic and task-specific spaces to apply differential regularization.
Disentanglement Architecture
Built upon a BERT backbone, IDBR splits the hidden representation into two distinct components. To enforce this separation, the model utilizes two auxiliary tasks as induction biases:
- Next Sentence Prediction (NSP): Trains the task-generic space by focusing on fundamental linguistic relationships.
- Task-ID Prediction: Trains the task-specific space by forcing it to learn features that distinguish one task from another.
Differential Regularization and Smart Replay
When transitioning to a new task, IDBR applies a strict penalty to the task-generic space to maintain stability, while allowing the task-specific space more room to adapt. Additionally, the model employs a Smart Replay mechanism. Instead of random sampling, it uses K-Means clustering to select a diverse 1% subset of representative examples from previous tasks, significantly reducing memory overhead compared to traditional replay methods.
Experimental Setup
The model was evaluated using five large-scale text classification benchmarks: AGNews (News), Yelp (Sentiment), Amazon (Sentiment), DBPedia (Wikipedia), and Yahoo! Answers (Q&A). These were tested in various sequences to measure long-term knowledge retention.
Results and Discussion
IDBR consistently outperformed state-of-the-art baselines, including MBPA++ and LAMOL. On a 5-task sequence, IDBR achieved an average accuracy of 76.3%, surpassing MBPA++ (74.3%).
Visualization of Disentanglement
The success of IDBR is rooted in its ability to effectively separate information. Visualization of the representation spaces confirms that the task-generic space clusters data based on linguistic commonalities, while the task-specific space clusters data according to the unique requirements of individual tasks.
Key Insights
- Stability through Genericity: Regularizing the generic space provides more significant stability gains than regularizing the specific space, highlighting the importance of preserving the linguistic "foundation."
- Synergy of Auxiliary Tasks: Ablation studies showed that both NSP and Task-ID prediction are necessary for effective disentanglement; removing either leads to a drop in accuracy.
Conclusion
IDBR introduces a robust method for continual learning in NLP by leveraging information disentanglement. By treating generic and specific knowledge differently, the model achieves high accuracy and stability with minimal memory requirements. Future work will explore the application of this strategy to more complex tasks such as sequence generation and labeling.
Summary updated