Out of Context: A New Clue for Context Modeling of Aspect-based Sentiment Analysis

Bowen Xing, Ivor W. Tsang

TL;DR

This paper introduces the "aspect-agnostic problem" in Aspect-Based Sentiment Analysis (ABSA), where standard context encoders (like LSTMs and BERT) fail to adequately focus on the specific aspect being analyzed, leading to the retention of irrelevant information. To combat this, the authors propose "aspect-aware" encoders (AALSTM, AAGCN, AABERTs) that explicitly incorporate aspect semantics into the context modeling process, significantly improving ABSA performance.

What Problem Does This Paper Address?

Aspect-Based Sentiment Analysis (ABSA) aims to determine the sentiment expressed towards a specific aspect within a piece of text. For example, in "The pizza was great, but the service was slow," we want to know the sentiment towards "pizza" (positive) and "service" (negative) separately. The core challenge in ABSA lies in effectively modeling the interaction between the text's context and the target aspect.

Existing methods often use context encoders like LSTMs or BERT to process the text and then employ attention mechanisms to link the context to the aspect. However, the authors identify a critical flaw: these standard encoders are "aspect-agnostic." They process the context without explicit guidance from the specific aspect being analyzed. This leads to two main issues:

  1. Discarding relevant information: Information crucial for understanding the sentiment towards the specific aspect might be overlooked or "forgotten" if it's not central to the overall sentence meaning.
  2. Retaining irrelevant information: Information that is relevant to other aspects or the general context, but not the target aspect, can be retained and even amplified, acting as noise and hindering accurate sentiment prediction for the target aspect.

This "aspect-agnostic problem" is particularly problematic in multi-aspect reviews where sentiment towards different aspects can be conflicting.

Methods

The paper proposes three main categories of "aspect-aware" context encoders, each building upon a standard architecture:

  • Aspect-Aware LSTM (AALSTM): This approach augments the standard LSTM with an "aspect-aware mechanism." This mechanism introduces three "aspect gates" (for input, forget, and output gates) that dynamically adjust how the aspect's semantics influence the LSTM's internal state updates. The aspect vector is used to regulate the flow of information, ensuring that aspect-relevant information is prioritized and aspect-irrelevant information is suppressed.
  • Aspect-Aware GCN (AAGCN): For Graph Convolutional Networks (GCNs), which model syntactic relationships, the authors introduce an "aspect-aware convolution gate." This gate controls how information is passed between nodes in the graph, ensuring that the aggregation of aspect-related information is prioritized and irrelevant information is filtered out.
  • Aspect-Aware BERT (AABERTs): Instead of altering BERT's internal architecture, the authors cleverly modify its input format. They propose three variants (AABERT1, AABERT2, AABERT3) that adjust segment embeddings and the use of the [SEP] token. The goal is to break down the strict separation between context and aspect in BERT's input, allowing BERT to better model the intra-sentence dependencies between the aspect and its related context words. This approach aims to make BERT "aspect-aware" without incurring additional computational costs.

Datasets

The models were evaluated on standard ABSA benchmark datasets:

  • SemEval-2014 Task 4: This includes datasets for both Aspect Term Sentiment Analysis (ATSA) and Aspect Category Sentiment Analysis (ACSA).
  • ATSA: Laptop reviews and Restaurant reviews.
  • ACSA: Restaurant reviews.

The paper also mentions using subsets of these datasets based on the number of aspects present in a review to analyze performance in multi-aspect scenarios.

Key Insights and Ideas

  • The "Aspect-Agnostic Problem": This is the paper's central contribution – identifying and formalizing the issue that standard context encoders are blind to the specific aspect they are supposed to analyze, leading to suboptimal performance.
  • Aspect as a "Clue Out of Context": The core idea is that the aspect itself should be an active participant in the context modeling process, not just a passive target for attention.
  • Dynamic Regulation of Information Flow: The proposed aspect-aware mechanisms (gates in LSTM, convolution gates in GCN, modified input in BERT) are designed to dynamically control what information is retained or discarded based on its relevance to the target aspect.
  • BERT's Input Format is Key: For BERT, the authors demonstrate that subtle changes to input formatting (segment embeddings, [SEP] token usage) can significantly improve its ability to capture aspect-specific dependencies without altering its core architecture or adding computational overhead.
  • Syntactic and Semantic Synergy: The work suggests that combining aspect-aware semantic modeling (AALSTM, AABERTs) with aspect-aware syntactic modeling (AAGCN) can lead to state-of-the-art results.

Key Results

  • Significant Performance Gains: The proposed aspect-aware models (AALSTM, AAGCN, AABERTs) consistently outperform their vanilla counterparts across various ABSA tasks (ATSA and ACSA) and datasets.
  • State-of-the-Art Achievements: When combined, the aspect-aware models achieve new state-of-the-art results on benchmark datasets, even surpassing models proposed years later. For instance, RAM combined with AAGCN and AABERT3 achieved SOTA on ATSA(ST2).
  • AABERTs' Efficiency: The AABERT variants achieve significant improvements over standard BERT with no extra computational cost, solely by modifying input formats.
  • Improved Handling of Neutral Sentiment and Multi-Aspect Reviews: AALSTM, in particular, shows a notable improvement in predicting neutral sentiment and handling reviews with multiple aspects, demonstrating its ability to better discern aspect-specific information.
  • Robustness in Cross-Domain Evaluation: AABERTs show better robustness for domain transfer compared to standard BERT.

Limitations and Future Work

  • Aspect Embedding Concatenation (AEC) Issue: The paper notes that aspect embedding concatenation (used in models like ATAE) can negatively impact the effectiveness of aspect-aware mechanisms like AALSTM, suggesting that the purity of input representations is crucial.
  • Multi-Aspect Interaction: While the proposed models alleviate the multi-aspect problem, the paper acknowledges that the adverse interactive effect between different aspects can still limit performance to some extent.
  • Future Work:
    • Exploring aspect-aware extensions for other architectures like Convolutional Neural Networks (CNNs) and Memory Networks.
  • Applying the aspect-aware concept to other NLP tasks, such as relation classification, where an "entity-pair agnostic" problem similar to the aspect-agnostic problem exists.
  • Further investigation into how to best model the relationship between context and aspect.

Summary updated

Abstract

Aspect-based sentiment analysis (ABSA) aims to predict the sentiment expressed in a review with respect to a given aspect. The core of ABSA is to model the interaction between the context and given aspect to extract aspect-related information. In prior work, attention mechanisms and dependency graph networks are commonly adopted to capture the relations between the context and given aspect. And the weighted sum of context hidden states is used as the final representation fed to the classifier. However, the information related to the given aspect may be already discarded and adverse information may be retained in the context modeling processes of existing models. Such a problem cannot be solved by subsequent modules due to two reasons. First, their operations are conducted on the encoder-generated context hidden states, whose value cannot be changed after the encoder. Second, existing encoders only consider the context while not the given aspect. To address this problem, we argue the given aspect should be considered as a new clue out of context in the context modeling process. As for solutions, we design three streams of aspect-aware context encoders: an aspect-aware LSTM, an aspect-aware GCN, and three aspect-aware BERTs. They are dedicated to generating aspect-aware hidden states which are tailored for the ABSA task. In these aspect-aware context encoders, the semantics of the given aspect is used to regulate the information flow. Consequently, the aspect-related information can be retained and aspect-irrelevant information can be excluded in the generated hidden states. We conduct extensive experiments on several benchmark datasets with empirical analysis, demonstrating the efficacies and advantages of our proposed aspect-aware context encoders.
BibTeX
@article{xing2022out,
  author = {Bowen Xing and Ivor W. Tsang},
  title = {{Out of Context: A New Clue for Context Modeling of Aspect-based Sentiment Analysis}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  url = {https://openreview.net/forum?id=BvhaDmsleT},
}
Figure