CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks

Zixuan Ke, Bing Liu, Hu Xu, Lei Shu

Abstract

This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks in a particular CL setting called domain incremental learning (DIL). Each task is from a different domain or product. The DIL setting is particularly suited to ASC because in testing the system needs not know the task/domain to which the test data belongs. To our knowledge, this setting has not been studied before for ASC. This paper proposes a novel model called CLASSIC. The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing. Experimental results show the high effectiveness of CLASSIC.

Introduction

Most sentiment analysis models are trained on a single domain (e.g., restaurant reviews) and struggle when moved to another (e.g., electronics). Continual Learning (CL) aims to learn a sequence of these tasks incrementally. However, two major hurdles exist: Catastrophic Forgetting (CF), where learning a new task causes the model to forget previous ones, and the Task ID Requirement. Many CL methods require knowing which domain a sentence belongs to at test time to select the correct sub-model. CLASSIC addresses these by focusing on Domain Incremental Learning (DIL), creating a system that works regardless of the domain without requiring task IDs during testing.

The CLASSIC Framework

CLASSIC (Continual and Linear Aspect Sentiment Classification) builds upon Adapter-BERT. Instead of tuning all of BERT's millions of parameters, the model inserts small, trainable "adapter" layers between BERT's fixed layers. This confines the "drift" that causes catastrophic forgetting to a small set of parameters, which are further protected using learnable "soft" binary Task Masks (TMs).

The CLASSIC architecture featuring Contrastive Ensemble Distillation (CED), Contrastive Supervised Learning (CSC), and Contrastive Knowledge Sharing (CKS).

The core innovation is a Contrastive Continual Learning framework consisting of three sub-systems:

  1. Contrastive Ensemble Distillation (CED): To fight forgetting, CLASSIC treats previous task models as "teachers." It uses a contrastive loss to ensure the current model's outputs remain similar to the outputs of previous models, distilling past knowledge into the new model.
  2. Contrastive Knowledge Sharing (CKS): This encourages "forward transfer" by using a task-based self-attention mechanism to identify previous tasks similar to the current one, aligning representations via contrastive learning.
  3. Contrastive Supervised Learning (CSC): This optimizes current task performance by ensuring instances with the same sentiment label are pulled together in the embedding space while different labels are pushed apart.

Experimental Setup

The researchers evaluated CLASSIC using a sequence of 19 ASC tasks derived from four major sources: HL5Domains (5 products), Liu3Domains (3 products), Ding9Domains (9 products), and SemEval14 (Laptop and Restaurant reviews). This represents a diverse array of domains with varying linguistic styles and vocabularies.

Results and Analysis

CLASSIC was compared against 46 baselines, including non-continual models and state-of-the-art CL methods like HAT, EWC, and LAMOL.

Performance comparison of CLASSIC against various baselines across non-continual and continual learning scenarios.

As shown in the results table, CLASSIC achieved an average Accuracy of 0.9022 and a Macro-F1 of 0.8512 across all 19 tasks, significantly outperforming the best baselines. The CED component proved vital for maintaining high performance on old tasks without task IDs, a scenario where other models like HAT failed. Ablation studies confirmed that removing any of the three contrastive components (CED, CKS, or CSC) led to a performance drop, validating the necessity of the integrated approach.

Conclusion and Future Work

CLASSIC demonstrates that contrastive learning can serve as a powerful bridge for knowledge transfer and distillation in CL. While the model is parameter-efficient, the use of three contrastive losses increases computational overhead during training. Future work may explore "curriculum learning" to address sequence sensitivity and test the framework's generalization to other NLP tasks like Named Entity Recognition (NER).

Summary updated

Abstract

This paper studies continual learning (CL) of a sequence of aspect sentiment classification(ASC) tasks in a particular CL setting called domain incremental learning (DIL). Each task is from a different domain or product. The DIL setting is particularly suited to ASC because in testing the system needs not know the task/domain to which the test data belongs. To our knowledge, this setting has not been studied before for ASC. This paper proposes a novel model called CLASSIC. The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing. Experimental results show the high effectiveness of CLASSIC.
BibTeX
@inproceedings{ke2021classic,
  author = {Zixuan Ke and Bing Liu and Hu Xu and Lei Shu},
  title = {{CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=AMlyB9N7tc},
}
Figure