A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis

Zehui Dai, Cheng Peng, Huajie Chen, Yadong Ding

Abstract

Aspect-Category Sentiment Analysis (ACSA) involves identifying sentiment polarities for predefined categories within a text. Traditional multi-task learning (MTL) approaches often suffer from catastrophic forgetting when new categories are introduced incrementally. We propose CNE-net, a framework that treats category names as semantic inputs rather than abstract labels. By utilizing a shared encoder and decoder architecture, CNE-net maintains performance on existing categories while learning new ones, achieving state-of-the-art results on standard benchmarks.

Introduction

In real-world Natural Language Processing (NLP) applications, the set of relevant aspect categories is rarely static. For instance, a sentiment analyzer for the automotive industry may need to incorporate "battery life" as electric vehicles gain market share. Most existing ACSA models employ a Multi-Task Learning (MTL) strategy with a shared encoder but separate decoders for each category. When these models are updated with new categories—a process known as incremental learning—the shared encoder often shifts its representation, leading to a significant performance drop on previously learned categories, a phenomenon called catastrophic forgetting.

Methodology: The CNE-net Framework

The proposed Category Name Embedding network (CNE-net) addresses these issues by incorporating category names directly into the model's input. This allows the model to leverage the semantic meaning of categories to identify relevant linguistic cues within a sentence.

Architecture

The input to the BERT encoder is structured as: [CLS] sentence [SEP] category1 [SEP] category2 ... [SEP] categoryN. This design enables the model to understand the relationship between the sentence and specific categories (e.g., linking "too expensive" to the "price" category).

Architecture of the CNE-net framework and its decoder variants

Unlike traditional models, CNE-net uses a single shared decoder. Since category information is embedded in the input features, the decoder learns a universal mapping from these features to sentiment polarities. The authors explored several attention-based decoder variants, identifying CNE-net-SEP-sent.-att. as the most effective. This variant employs a two-step attention mechanism: first, the category's separator token ([SEP]) attends to the sentence; then, the resulting representation attends back to the category name to generate highly specific features.

Experimental Evaluation

The framework was evaluated on two benchmark datasets: SemEval-2014 Task 4 (restaurant reviews) and SentiHood (location-based reviews). To test incremental learning capabilities, the authors created SemEval14-Task-inc and Sentihood-inc, where categories are partitioned into "source" (previously learned) and "target" (newly introduced) sets.

Results and Discussion

CNE-net achieved state-of-the-art (SOTA) performance across both benchmarks. On the SemEval-14 dataset, it reached a sentiment accuracy of 87.1% for 4-way classification, outperforming existing models like QA-B and standard MTL.

Incremental Learning Stability

The primary strength of CNE-net is its resilience to catastrophic forgetting. In incremental tests, standard MTL models experienced drastic performance declines on source categories. In contrast, CNE-net maintained stable performance on old categories even after training on new ones.

Incremental learning performance on SemEval14-Task4-inc and Sentihood-inc datasets

The results demonstrate that sharing the decoder is critical for incremental learning. Because the decoder weights are universal, fine-tuning on new categories refines the general sentiment-mapping ability rather than overwriting category-specific logic. Furthermore, the model shows a strong ability to capture implicit sentiment, correctly associating descriptive phrases with categories even when the category name is absent from the review text.

Conclusion

CNE-net provides a robust solution for incremental ACSA by treating categories as semantic entities. By sharing both encoders and decoders, the model effectively mitigates catastrophic forgetting. Future work may explore the zero-shot learning potential of this architecture, as the model's understanding of category names could allow it to predict sentiments for entirely unseen categories.

Summary updated

BibTeX
@inproceedings{dai2020multi,
  author = {Zehui Dai and Cheng Peng and Huajie Chen and Yadong Ding},
  title = {{A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.565},
}
Figure