Abstract
Aspect-based sentiment analysis (ABSA) and Targeted ASBA (TABSA) allow finer-grained inferences about sentiment to be drawn from the same text, depending on context. For example, a given text can have different targets (e.g., neighborhoods) and different aspects (e.g., price or safety), with different sentiment associated with each target-aspect pair. In this paper, we investigate whether adding context to self-attention models improves performance on (T)ABSA. We propose two variants of Context-Guided BERT (CG-BERT) that learn to distribute attention under different contexts. We first adapt a context-aware Transformer to produce a CG-BERT that uses context-guided softmax-attention. Next, we propose an improved Quasi-Attention CG-BERT model that learns a compositional attention that supports subtractive attention. We train both models with pretrained BERT on two (T)ABSA datasets: SentiHood and SemEval-2014 (Task 4). Both models achieve new state-of-the-art results with our QACG-BERT model having the best performance. Furthermore, we provide analyses of the impact of context in the our proposed models. Our work provides more evidence for the utility of adding context-dependencies to pretrained self-attention-based language models for context-based natural language tasks.
Introduction
Standard sentiment analysis typically assigns a single polarity to an entire sentence. However, real-world reviews are often more complex, containing multiple targets and aspects with varying sentiments. This complexity necessitates Aspect-Based Sentiment Analysis (ABSA) and its more granular counterpart, Targeted ABSA (TABSA). In TABSA, the goal is to identify the sentiment for a specific entity (target) regarding a specific attribute (aspect).
While models like BERT have significantly advanced NLP, they are often treated as "black boxes" in sentiment tasks, where context is simply appended to the input. This paper argues that the model’s internal attention mechanism should be explicitly guided by the context—the specific target and aspect being queried—to focus on relevant tokens.
Methodology: Context-Guided Architectures
The authors propose two modifications to the standard Transformer architecture to integrate context directly into the self-attention mechanism.
Context-Guided BERT (CG-BERT)
CG-BERT modifies the Query ($Q$) and Key ($K$) matrices. It uses a gating mechanism with a $tanh$ activation to integrate a "context matrix" representing the target-aspect pair. This allows the model to shift its attention weights dynamically based on the specific query.
Quasi-Attention CG-BERT (QACG-BERT)
The more advanced variant, QACG-BERT, introduces Quasi-Attention. Unlike standard softmax attention, which is strictly additive (weights between 0 and 1), Quasi-Attention allows for negative weights. This enables "subtractive attention," allowing the model to explicitly "de-attend" to or ignore words that might be misleading or irrelevant for a specific context.
Experimental Results
The models were evaluated on two benchmarks: SentiHood (London neighborhood reviews) and SemEval-2014 Task 4 (restaurant reviews).
The QACG-BERT model established new state-of-the-art (SOTA) results:
- SentiHood: Achieved 80.9% Strict Accuracy for aspect categorization and 93.8% Accuracy for sentiment classification.
- SemEval-2014: Achieved an F1-score of 92.64% for aspect categorization and 86.8% Accuracy for 4-class sentiment.
Key Insights
- Subtractive Attention: The ability to assign negative weights is crucial. For instance, the word "nice" is generally positive, but in the context of "price," a "nice and expensive" location is negative. QACG-BERT can subtract the influence of "nice" when it doesn't align with the aspect.
- Context-Specific Saliency: Gradient sensitivity analysis confirmed that the models focus on different keywords (e.g., "rent" for price vs. "location" for transit) depending on the aspect queried.
- Initialization Strategy: To preserve pretrained knowledge, new weights were initialized to very small values, allowing the model to gradually learn context integration during fine-tuning.
Conclusion
This work demonstrates that explicitly guiding self-attention with context significantly improves performance on fine-grained sentiment tasks. While adding these layers increases parameter counts, the gains in accuracy and the ability to perform subtractive attention suggest that context-aware mechanisms are a promising direction for complex NLU tasks.
Summary updated