Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis

Zhengxuan Wu, Desmond C. Ong

Abstract

Aspect-based sentiment analysis (ABSA) and Targeted ASBA (TABSA) allow finer-grained inferences about sentiment to be drawn from the same text, depending on context. For example, a given text can have different targets (e.g., neighborhoods) and different aspects (e.g., price or safety), with different sentiment associated with each target-aspect pair. In this paper, we investigate whether adding context to self-attention models improves performance on (T)ABSA. We propose two variants of Context-Guided BERT (CG-BERT) that learn to distribute attention under different contexts. We first adapt a context-aware Transformer to produce a CG-BERT that uses context-guided softmax-attention. Next, we propose an improved Quasi-Attention CG-BERT model that learns a compositional attention that supports subtractive attention. We train both models with pretrained BERT on two (T)ABSA datasets: SentiHood and SemEval-2014 (Task 4). Both models achieve new state-of-the-art results with our QACG-BERT model having the best performance. Furthermore, we provide analyses of the impact of context in the our proposed models. Our work provides more evidence for the utility of adding context-dependencies to pretrained self-attention-based language models for context-based natural language tasks.

Introduction

Standard sentiment analysis typically evaluates a sentence as a single unit, assigning a global polarity. However, real-world text often contains nuanced opinions regarding different entities and categories. This has led to the development of Aspect-Based Sentiment Analysis (ABSA), which identifies sentiment for specific categories (e.g., food vs. service), and Targeted ABSA (TABSA), which handles multiple entities within the same context.

Example of Targeted Aspect-Based Sentiment Analysis (TABSA) where different targets (LOC1, LOC2) and aspects (price, safety) in a single sentence yield different sentiment polarities.

As shown in the figure above, a single sentence can contain conflicting sentiments depending on the target and aspect being queried. While BERT has revolutionized NLP, it is often used as a "black box" for these tasks. This research argues that the BERT architecture itself should be modified to be context-aware, allowing its self-attention mechanism to shift focus dynamically based on the specific target and aspect under evaluation.

Proposed Methods

The researchers propose two variants of "Context-Guided" BERT that modify the internal self-attention layers to incorporate target and aspect information.

CG-BERT (Context-Guided BERT)

This model modifies the internal Query ($Q$) and Key ($K$) matrices of BERT's self-attention layers. It introduces a "gating unit" that incorporates a context matrix representing the specific target and aspect. This allows the model to emphasize or de-emphasize parts of the sentence based on the context provided during the forward pass.

The CG-BERT architecture introduces context-aware matrices to the Query and Key components of the self-attention layer to guide the attention mechanism.

QACG-BERT (Quasi-Attention Context-Guided BERT)

QACG-BERT introduces a more novel approach called Quasi-Attention. Standard attention mechanisms use a softmax function, which forces all weights to be positive and sum to one. This implies that every word can only add to the final representation. Quasi-attention allows for subtractive attention, enabling the model to effectively "de-attend" from words that might be misleading in a specific context. For example, in a review discussing both "price" and "safety," the model can subtract the influence of safety-related words when specifically evaluating the price aspect.

Experimental Results

The models were evaluated on two major benchmarks: SentiHood (London neighborhood reviews) and SemEval-2014 Task 4 (restaurant reviews).

QACG-BERT achieved new State-of-the-Art (SOTA) performance across both datasets:

  • SentiHood: Achieved a Strict Accuracy of 80.9% and a Macro-F1 of 89.7% for aspect categorization.
  • SemEval-2014: Surpassed previous SOTA models in both aspect categorization and sentiment classification across multiple evaluation classes.

Key Insights

  1. The Power of Subtraction: The ability to assign negative weights via quasi-attention proved crucial. It allows the model to ignore generally positive but contextually irrelevant words (e.g., ignoring "nice" when evaluating "price").
  2. Context-Dependent Importance: Gradient sensitivity analysis confirmed that QACG-BERT identifies different "important" words for the same sentence depending on the aspect. For "price," it focuses on terms like "costs"; for "transit," it shifts focus to "location."
  3. Smart Initialization: To preserve the power of pretrained BERT, new context-aware layers were initialized with very small random weights, allowing the model to gradually learn context integration during fine-tuning without disrupting established language patterns.

Conclusion and Future Work

By making the self-attention mechanism context-aware, QACG-BERT provides a more robust framework for fine-grained sentiment analysis. While the architectural complexity and parameter count increase slightly (from ~110M to ~124M), the performance gains justify the overhead. Future research may apply this context-guided approach to other query-based NLP tasks, such as document summarization and complex question answering.

Summary updated

Abstract

Aspect-based sentiment analysis (ABSA) and Targeted ASBA (TABSA) allow finer-grained inferences about sentiment to be drawn from the same text, depending on context. For example, a given text can have different targets (e.g., neighborhoods) and different aspects (e.g., price or safety), with different sentiment associated with each target-aspect pair. In this paper, we investigate whether adding context to self-attention models improves performance on (T)ABSA. We propose two variants of Context-Guided BERT (CG-BERT) that learn to distribute attention under different contexts. We first adapt a context-aware Transformer to produce a CG-BERT that uses context-guided softmax-attention. Next, we propose an improved Quasi-Attention CG-BERT model that learns a compositional attention that supports subtractive attention. We train both models with pretrained BERT on two (T)ABSA datasets: SentiHood and SemEval-2014 (Task 4). Both models achieve new state-of-the-art results with our QACG-BERT model having the best performance. Furthermore, we provide analyses of the impact of context in the our proposed models. Our work provides more evidence for the utility of adding context-dependencies to pretrained self-attention-based language models for context-based natural language tasks.
BibTeX
@inproceedings{wu2021context,
  author = {Zhengxuan Wu and Desmond C. Ong},
  title = {{Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Sc6v9mpKAQd},
}
Figure