Research on Multi-head Self-attention Aspect Term Extraction with Multi-level Features Encoding

Penghui Zhang, Peng Yang, Yang PengAdvances 2022 · DOI · OpenAlex

Abstract

With the development of social media and e-commerce, aspect-based sentiment analysis technique is more and more urgently needed. Aspect term extraction is an important part of aspect-based sentiment analysis, and the quality of the extracted aspect words directly determines the quality of aspect-based sentiment analysis. To solve the problems of current research, in this paper, we design and implement a multi-head self-attention aspect term extraction fusing with multilevel features model (MSA-FMF). In addition, we conduct experiments on three public datasets, and the proposed model outperforms the comparison models, proving the effectiveness of our model.

Introduction

Aspect-Based Sentiment Analysis (ABSA) is a fine-grained sentiment analysis task that aims to identify the sentiment expressed towards specific aspects of an entity. The first and most critical step in this process is Aspect Term Extraction (ATE), which involves identifying the specific "aspect" words in a sentence (e.g., "service" in a restaurant review or "battery life" for a laptop).

Examples of aspect term extraction in different contexts.

Existing ATE research faces two primary challenges:

  1. Feature Neglect: Many models overlook character-level features and fine-grained semantic associations, leading to fragmented extraction.
  2. Contextual Mismatch: Models often fail to learn the relationship between an aspect word and its surrounding context, incorrectly identifying words based on frequency rather than their functional role in a sentence.

Methodology: The MSA-FMF Model

To address these issues, we propose the MSA-FMF (Multi-head Self-attention Aspect term extraction fusing with Multi-level Features) model. The architecture consists of four distinct layers designed to capture both local and global dependencies.

Architecture of the MSA-FMF model showing the multi-level feature encoding, context encoding, global context extraction, and sequence decoding layers.

Multi-level Features Encoding Layer

This layer utilizes a pre-trained RoBERTa-base model to extract word, position, and segment embeddings. To capture sub-word information and handle rare words or morphological variations, a Character Encoding Layer using a BiLSTM is integrated. This results in a representation that fuses four different granularities of data.

Context Encoding Layer

A bidirectional Long Short-Term Memory (BiLSTM) network processes the fused features to integrate sentence-level contextual information into the vector representations.

Global Context Information Extraction Layer

At the core of the model is a Multi-head Self-attention mechanism (using 4 heads) that learns long-range dependencies. This layer specifically addresses the "mismatch" problem by allowing the model to evaluate the entire sentence context when determining if a word functions as an aspect.

Sequence Decoding Layer

Finally, a Conditional Random Field (CRF) performs sequence labeling using the IOB (Begin, Inside, Outside) format, ensuring globally optimal label predictions for the sentence structure.

Experimental Setup

The MSA-FMF model was evaluated on three benchmark datasets:

  • ACL14 Twitter: Social media posts.
  • SemEval2014 Restaurant: Reviews from the food service industry.
  • SemEval2014 Laptop: Hardware technical reviews.

All datasets were annotated using the IOB method and split into an 8:2 ratio for training and validation.

Results and Discussion

The MSA-FMF model demonstrated superior performance compared to strong baselines such as LSTM-CRF, BiLSTM-CNN, and BERT-PT.

  • Laptop Dataset: Achieved an F1-score of 86.16, outperforming the previous best (BAT) by 0.59 percentage points.
  • Restaurant Dataset: Achieved an F1-score of 87.90, a significant improvement of 2.59 percentage points over the BiDTreeCRF model.
  • Twitter Dataset: Showed dominant performance with a Precision of 98.23%, Recall of 98.68%, and an F1-score of 98.45, surpassing the BERT-PT model by 2.35 points.

Key findings indicate that hybrid granularity (combining word and character embeddings) and global attention refinement are essential for reducing contextual errors. The choice of RoBERTa as a backbone also provided a more robust embedding foundation than standard BERT.

Conclusion

The MSA-FMF model effectively addresses the limitations of feature neglect and contextual mismatch in Aspect Term Extraction. By fusing multi-level features with multi-head self-attention, the model achieves state-of-the-art results across diverse domains. Future work may explore optimizing the computational efficiency of this multi-layered architecture.

Summary updated

Abstract

With the development of social media and e-commerce, aspect-based sentiment analysis technique is more and more urgently needed. Aspect term extraction is an important part of aspect-based sentiment analysis, and the quality of the extracted aspect words directly determines the quality of aspect-based sentiment analysis. To solve the problems of current research, in this paper, we design and implement a multi-head self-attention aspect term extraction fusing with multilevel features model (MSA-FMF). In addition, we conduct experiments on three public datasets, and the proposed model outperforms the comparison models, proving the effectiveness of our model.
BibTeX
@inproceedings{zhang2022research,
  author = {Penghui Zhang and Peng Yang and Yang Peng},
  title = {{Research on Multi-head Self-attention Aspect Term Extraction with Multi-level Features Encoding}},
  booktitle = {Advances},
  year = {2022},
  doi = {10.2991/978-94-6463-046-6\_45},
}
Figure