Abstract
Aspect-Category Sentiment Analysis (ACSA) typically relies on multi-task learning (MTL) architectures that suffer from "catastrophic forgetting" when new categories are introduced. We present CNE-net, a framework that treats category names as semantic features rather than simple indices. By sharing both the encoder and decoder across all categories, CNE-net enables incremental learning of new sentiment categories without degrading performance on previously learned ones.
Introduction
In ACSA, models identify sentiment toward predefined categories such as "food" or "service." While current MTL models achieve high performance, they are often brittle due to their reliance on category-specific decoders. This architectural choice leads to two primary issues: inflexibility in adding new categories and catastrophic forgetting, where fine-tuning on new tasks causes the shared encoder to drift away from the requirements of original, static decoders. The authors aim to make ACSA models capable of incremental learning—adding new categories over time while maintaining high accuracy on the old ones.
Methodology: The CNE-net Architecture
The core innovation of this work is the Category Name Embedding network (CNE-net). By embedding category names (e.g., "price," "transit-location") directly into the input sequence, the model leverages the semantic relationships between category labels and sentence content.
Input Structure and Encoder
CNE-net utilizes a BERT-base encoder. The input is formatted as a single sequence:
[CLS] + Sentence + [SEP] + Category_1 + [SEP] + Category_2 ... [SEP]
Decoder Variants
The authors proposed three decoder architectures to extract category-specific features from the BERT output:
- CNE-net-SEP: Extracts features from the
[SEP]token following each category name. - CNE-net-CLS-att: Uses the
[CLS]token to attend to words within the category name. - CNE-net-SEP-sent.-att: A dual-attention mechanism where a category's
[SEP]token first attends to the sentence to create a category-aware vector, which then attends back to the category name words.
Shared Weights and Incremental Learning
Unlike traditional MTL, CNE-net shares the final softmax classifier weights across all categories. This unified structure ensures that the entire pipeline remains synchronized during updates, effectively mitigating forgetting. Because the category identity is embedded in the input text itself, a single decoder can learn to distinguish between tasks.
Experimental Results
The model was evaluated on SemEval-2014 Task 4 and SentiHood. To test incremental learning, the authors created "incremental" versions of these datasets (SemEval14-Task-inc and Sentihood-inc), splitting categories into "source" (old) and "target" (new) sets.
State-of-the-Art Performance
CNE-net-SEP-sent.-att achieved state-of-the-art results on standard benchmarks, reaching a 92.27 F1 for category extraction and 87.1% accuracy for 4-way sentiment classification on SemEval14. It outperformed previous models, including BERT-pair-QA-B, on both standard datasets.
Mitigating Catastrophic Forgetting
The most significant finding was the model's stability during incremental updates. While standard MTL models saw performance on source categories drop significantly (e.g., from ~90 F1 to ~70 F1), CNE-net maintained nearly stable performance, dropping by less than 0.5% in most metrics.
Discussion and Future Work
By treating category names as semantic features, CNE-net allows the model to understand why specific phrases relate to specific categories. This approach not only improves accuracy but also provides a path toward zero-shot learning, where the model could potentially predict sentiment for entirely unseen categories based on their names. Future research will address the sequence length limitations imposed by BERT's 512-token limit when dealing with very large category sets and explore the computational efficiency of the dual-attention mechanism.
Summary updated