A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis

Zehui Dai, Cheng Peng, Huajie Chen, Yadong Ding

Abstract

Aspect-Category Sentiment Analysis (ACSA) typically relies on multi-task learning (MTL) architectures that suffer from "catastrophic forgetting" when new categories are introduced. We present CNE-net, a framework that treats category names as semantic features rather than simple indices. By sharing both the encoder and decoder across all categories, CNE-net enables incremental learning of new sentiment categories without degrading performance on previously learned ones.

Introduction

In ACSA, models identify sentiment toward predefined categories such as "food" or "service." While current MTL models achieve high performance, they are often brittle due to their reliance on category-specific decoders. This architectural choice leads to two primary issues: inflexibility in adding new categories and catastrophic forgetting, where fine-tuning on new tasks causes the shared encoder to drift away from the requirements of original, static decoders. The authors aim to make ACSA models capable of incremental learning—adding new categories over time while maintaining high accuracy on the old ones.

Methodology: The CNE-net Architecture

The core innovation of this work is the Category Name Embedding network (CNE-net). By embedding category names (e.g., "price," "transit-location") directly into the input sequence, the model leverages the semantic relationships between category labels and sentence content.

Input Structure and Encoder

CNE-net utilizes a BERT-base encoder. The input is formatted as a single sequence: [CLS] + Sentence + [SEP] + Category_1 + [SEP] + Category_2 ... [SEP]

Decoder Variants

The authors proposed three decoder architectures to extract category-specific features from the BERT output:

  1. CNE-net-SEP: Extracts features from the [SEP] token following each category name.
  2. CNE-net-CLS-att: Uses the [CLS] token to attend to words within the category name.
  3. CNE-net-SEP-sent.-att: A dual-attention mechanism where a category's [SEP] token first attends to the sentence to create a category-aware vector, which then attends back to the category name words.
Architecture of the three proposed CNE-net decoder variants.

Shared Weights and Incremental Learning

Unlike traditional MTL, CNE-net shares the final softmax classifier weights across all categories. This unified structure ensures that the entire pipeline remains synchronized during updates, effectively mitigating forgetting. Because the category identity is embedded in the input text itself, a single decoder can learn to distinguish between tasks.

Experimental Results

The model was evaluated on SemEval-2014 Task 4 and SentiHood. To test incremental learning, the authors created "incremental" versions of these datasets (SemEval14-Task-inc and Sentihood-inc), splitting categories into "source" (old) and "target" (new) sets.

State-of-the-Art Performance

CNE-net-SEP-sent.-att achieved state-of-the-art results on standard benchmarks, reaching a 92.27 F1 for category extraction and 87.1% accuracy for 4-way sentiment classification on SemEval14. It outperformed previous models, including BERT-pair-QA-B, on both standard datasets.

Mitigating Catastrophic Forgetting

The most significant finding was the model's stability during incremental updates. While standard MTL models saw performance on source categories drop significantly (e.g., from ~90 F1 to ~70 F1), CNE-net maintained nearly stable performance, dropping by less than 0.5% in most metrics.

Incremental learning performance comparison on SemEval14 and SentiHood benchmarks.

Discussion and Future Work

By treating category names as semantic features, CNE-net allows the model to understand why specific phrases relate to specific categories. This approach not only improves accuracy but also provides a path toward zero-shot learning, where the model could potentially predict sentiment for entirely unseen categories based on their names. Future research will address the sequence length limitations imposed by BERT's 512-token limit when dealing with very large category sets and explore the computational efficiency of the dual-attention mechanism.

Summary updated

BibTeX
@inproceedings{dai2020multi,
  author = {Zehui Dai and Cheng Peng and Huajie Chen and Yadong Ding},
  title = {{A Multi-Task Incremental Learning Framework with Category Name Embedding for Aspect-Category Sentiment Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.565},
}
Figure