CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks

Zixuan Ke, Bing Liu, Hu Xu, Lei Shu

Introduction

Sentiment analysis models typically operate on static datasets, failing to adapt as new products and domains emerge. Continual Learning (CL) addresses this by enabling models to learn tasks sequentially. However, CL faces two primary challenges: Catastrophic Forgetting (CF), where new knowledge overwrites old, and the Task-ID requirement, which necessitates knowing the domain source during inference. This paper introduces CLASSIC (Continual and Learning for Aspect Sentiment Classification), a framework designed for Domain Incremental Learning (DIL) in Aspect Sentiment Classification (ASC).

Methodology

CLASSIC utilizes a frozen BERT backbone augmented with Adapters—small, trainable layers that preserve core weights while learning task-specific features. The framework's novelty lies in its three contrastive learning objectives:

  • Contrastive Ensemble Distillation (CED): Prevents forgetting by aligning the current model's outputs with an ensemble of previous "teacher" models using contrastive loss.
  • Contrastive Knowledge Sharing (CKS): Employs a task-based self-attention mechanism to identify and pull relevant knowledge from similar previous domains.
  • Contrastive Supervised Learning (CSC): Clusters embeddings with the same sentiment labels while separating different ones to refine internal representations.
The CLASSIC framework architecture, illustrating the Contrastive Ensemble Distillation (CED), Contrastive Supervised Learning (CSC), and Contrastive Knowledge Sharing (CKS) modules.

Additionally, Task Masks (TMs) are employed to protect neurons associated with previous tasks, ensuring they remain unmodified during subsequent training phases.

Experimental Setup

The model was evaluated on a sequence of 19 ASC tasks sourced from four major datasets: HL5Domains, Liu3Domains, Ding9Domains, and SemEval14. This extensive benchmark covers diverse products ranging from electronics (e.g., Nokia 6610, Canon G3) to software (Norton) and services (Restaurants). The evaluation compared CLASSIC against 46 baselines, including state-of-the-art CL methods like HAT, EWC, and LAMOL.

Results and Discussion

CLASSIC achieves state-of-the-art performance across the 19-task sequence, demonstrating the effectiveness of the contrastive approach in a task-ID-free setting.

Performance comparison across 19 tasks, showing CLASSIC outperforming state-of-the-art baselines in both Accuracy and Macro-F1.

Key findings include:

  • Superior Accuracy: CLASSIC achieved an average Accuracy of 0.9022 and a Macro-F1 of 0.8512, significantly outperforming LAMOL (0.8891 Accuracy).
  • Effective Knowledge Transfer: The framework successfully achieved both positive forward transfer (leveraging old knowledge for new tasks) and backward transfer (refining old task performance through new learning).
  • Robustness of Adapters: The study confirms that Adapters provide a more stable "workspace" for new knowledge compared to standard BERT fine-tuning, which often captures brittle, task-specific features.

Conclusion

CLASSIC proves that combining Adapters with contrastive learning objectives effectively mitigates catastrophic forgetting in domain-incremental settings. By treating the final model as a "universal" classifier, it eliminates the need for Task-IDs at inference, making it highly suitable for real-world sentiment analysis applications. Future work may extend this architecture to extraction-based tasks and other NLP domains like Named Entity Recognition.

Summary updated

Abstract

This paper studies continual learning (CL) of a sequence of aspect sentiment classification(ASC) tasks in a particular CL setting called domain incremental learning (DIL). Each task is from a different domain or product. The DIL setting is particularly suited to ASC because in testing the system needs not know the task/domain to which the test data belongs. To our knowledge, this setting has not been studied before for ASC. This paper proposes a novel model called CLASSIC. The key novelty is a contrastive continual learning method that enables both knowledge transfer across tasks and knowledge distillation from old tasks to the new task, which eliminates the need for task ids in testing. Experimental results show the high effectiveness of CLASSIC.
BibTeX
@inproceedings{ke2021classic,
  author = {Zixuan Ke and Bing Liu and Hu Xu and Lei Shu},
  title = {{CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=AMlyB9N7tc},
}
Figure