Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks

Zixuan Ke, Hu Xu, Bing Liu

TL;DR

This paper introduces B-CL, a novel approach to adapt BERT for continual learning of Aspect Sentiment Classification (ASC) tasks. B-CL effectively transfers knowledge between related tasks and prevents forgetting of previously learned information, significantly outperforming existing methods.

What Problem Does This Paper Address?

Continual Learning (CL) aims to train models incrementally on a sequence of tasks without forgetting previously learned knowledge. While CL has been explored for general sentiment classification, it remains a challenge for Aspect Sentiment Classification (ASC) tasks. ASC involves classifying the sentiment towards a specific aspect within a sentence, making it more nuanced. Existing CL methods often struggle with two key issues: (1) effectively transferring knowledge from previous tasks to improve learning on new tasks, and (2) preventing "catastrophic forgetting," where learning a new task degrades performance on older tasks. This paper addresses the need for a CL system specifically designed for ASC that can both leverage past knowledge and retain it.

Methods

The proposed method, B-CL, builds upon BERT, a powerful pre-trained language model, and incorporates a novel component called the Continual Learning Adapter (CLA). CLA is designed to address the two core CL challenges:

  • Knowledge Transfer: CLA utilizes a Knowledge Sharing Module (KSM) inspired by capsule networks. KSM employs task capsules and a dynamic routing algorithm to identify and group similar tasks and their shared knowledge. This allows for the transfer of relevant features and knowledge from previously learned tasks to the new task.
  • Forgetting Avoidance: CLA also includes a Task Specific Module (TSM) that uses task masks. These masks selectively protect neurons important for previous tasks from being updated when learning a new task. This mechanism helps isolate task-specific knowledge and prevent catastrophic forgetting.

B-CL integrates CLA into each transformer layer of BERT, allowing for end-to-end training. The system aims to achieve both forward transfer (improving new task learning) and backward transfer (improving performance on old tasks after learning new ones).

Datasets

The paper uses a collection of 19 Aspect Sentiment Classification (ASC) datasets, representing reviews of 19 different products. These datasets are sourced from:

  • HL5Domains (5 products)
  • Liu3Domains (3 products)
  • Ding9Domains (9 products)
  • SemEval14 (2 products: laptop and restaurant)

The authors split these datasets into training, validation, and testing sets, with specific details provided for each source. They do not introduce new datasets but rather leverage existing ones to create a challenging CL benchmark for ASC.

Key Insights and Ideas

  • BERT's Limitation in CL: Fine-tuned BERT, while excellent for single-task learning, performs poorly in a continual learning setting. This is because it learns highly task-specific information that is difficult to transfer and leads to significant forgetting.
  • Capsule Networks for Task Similarity: The paper creatively leverages capsule networks and dynamic routing not for their original classification purpose, but to identify and group similar tasks based on their learned features. This is a novel application of capsule networks in the CL domain.
  • Dual-Module Adapter for CL: The CLA's two-pronged approach (KSM for sharing, TSM for protection) is crucial. It highlights that effective CL requires both mechanisms to work in tandem.
  • Task Masks for Targeted Protection: The use of differentiable task masks provides a fine-grained way to protect specific neurons, allowing for more precise control over knowledge retention compared to broader regularization methods.
  • ASC's Unique CL Challenges: The paper emphasizes that ASC tasks, while related, have specific nuances that require tailored CL solutions, differentiating it from general sentiment classification CL.

Key Results

  • Superior Performance: B-CL significantly outperforms all baselines, including non-continual learning (NL) methods and various state-of-the-art continual learning systems, in both accuracy and Macro-F1 scores.
  • Forward Transfer Effectiveness: B-CL demonstrates highly effective forward transfer, meaning it significantly improves the learning of new tasks by leveraging knowledge from previous ones.
  • Backward Transfer Improvement: While the primary focus is on forward transfer and forgetting avoidance, B-CL also shows slight improvements in backward transfer, meaning performance on older tasks is maintained or even improved after learning new tasks.
  • Ablation Study Validation: An ablation study confirmed that each component of B-CL (KSM and TSM) is essential for its overall effectiveness, with the full B-CL system achieving the best results.

Limitations and Future Work

  • Hyperparameter Sensitivity: While not explicitly detailed as a limitation, the paper mentions the use of hyperparameters like the number of knowledge sharing capsules and the annealing strategy for task masks, suggesting that tuning these could be important.
  • Computational Cost: Adapting BERT with CLA likely increases computational complexity compared to simpler models, though this is not directly quantified as a limitation.
  • Future Work: The authors suggest that their work provides a strong foundation for future research in CL for ASC. Potential future directions could involve exploring more sophisticated capsule network architectures within CLA, investigating different task sequencing strategies, and applying B-CL to other fine-grained NLP tasks that require continual learning.

Summary updated

BibTeX
@inproceedings{ke2021adapting,
  author = {Zixuan Ke and Hu Xu and Bing Liu},
  title = {{Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=vEQfYgpJGW},
}
Figure