Abstract
This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks. Although some CL techniques have been proposed for document sentiment classification, we are not aware of any CL work on ASC. A CL system that incrementally learns a sequence of ASC tasks should address the following two issues: (1) transfer knowledge learned from previous tasks to the new task to help it learn a better model, and (2) maintain the performance of the models for previous tasks so that they are not forgotten. This paper proposes a novel capsule network based model called B-CL to address these issues. B-CL markedly improves the ASC performance on both the new task and the old tasks via forward and backward knowledge transfer. The effectiveness of B-CL is demonstrated through extensive experiments.
Introduction
Aspect Sentiment Classification (ASC) is a fine-grained sentiment analysis task that identifies the sentiment expressed toward a specific aspect or feature of an entity (e.g., "The camera is great, but the battery sucks"). In real-world applications, models often need to learn from a continuous stream of tasks—such as new product domains—while adhering to privacy constraints that prevent storing data from previous tasks. This scenario, known as Continual Learning (CL), is plagued by "catastrophic forgetting," where learning a new task significantly degrades performance on previously learned ones.
Existing CL methods often struggle with ASC because tasks are highly related—sharing common sentiment lexicons—yet require domain-specific nuances. An ideal CL system for ASC must achieve two goals: forward transfer (leveraging prior knowledge for new tasks) and forgetting avoidance (preserving performance on old tasks).
Methodology: The B-CL Architecture
The proposed B-CL model integrates a Continual Learning Adapter (CLA) into a frozen BERT backbone. By keeping the massive BERT parameters fixed and only training the small CLA modules, the model minimizes weight drift and catastrophic forgetting.
As shown in the architecture diagram below, the CLA consists of two primary components:
- Knowledge Sharing Module (KSM): Utilizing Capsule Networks, the KSM employs dynamic routing to group similar features across tasks. This allows the model to identify and transfer shared sentiment knowledge across different product domains.
- Task Specific Module (TSM): The TSM uses Task Masks (hard attention) to protect neurons critical for specific tasks. During training on a new task, gradients for these "protected" neurons are masked, preventing them from being overwritten.
Experimental Evaluation
The model was evaluated on a sequence of 19 ASC tasks derived from four major sources: HL5Domains, Liu3Domains, Ding9Domains, and SemEval 2014. These datasets cover diverse product categories, from vacuum cleaners to laptops. The experiments simulated a true continual learning environment by feeding these tasks to the model in random sequences.
Results and Discussion
B-CL demonstrated superior performance compared to 18 baselines, including prominent CL methods like HAT, EWC, and OWM. As shown in the results table, B-CL achieves the highest accuracy and Macro-F1 scores across the task sequence.
A key finding is the presence of backward transfer. Unlike many CL models that merely aim to minimize forgetting, B-CL's shared knowledge space allows it to improve performance on older tasks as it learns related new tasks. This is evidenced by the performance gain of the full B-CL model over the "B-CL (forward)" variant. Ablation studies further confirmed that both the KSM and TSM are essential; removing either component leads to a significant drop in performance.
Conclusion
B-CL effectively addresses the challenges of continual learning for ASC by combining the strengths of capsule networks for knowledge sharing and task masking for weight protection. While the current model relies on Task IDs during testing (Task Incremental Learning), future work will explore Class Incremental Learning scenarios where the model must infer the task context autonomously.
Summary updated