Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification

Xiaochen Hou, Jing Huang, Guangtao Wang, Peng Qi, Xiaodong He, Bowen Zhou2021 · DOI

Abstract

Recent work on aspect-level sentiment classification has employed Graph Convolutional Networks (GCN) over dependency trees to learn interactions between aspect terms and opinion words. In some cases, the corresponding opinion words for an aspect term cannot be reached within two hops on dependency trees, which requires more GCN layers to model. However, GCNs often achieve the best performance with two layers, and deeper GCNs do not bring any additional gain. Therefore, we design a novel selective attention based GCN model. On one hand, the proposed model enables the direct interaction between aspect terms and context words via the self-attention operation without the distance limitation on dependency trees. On the other hand, a top-k selection procedure is designed to locate opinion words by selecting k context words with the highest attention scores. We conduct experiments on several commonly used benchmark datasets and the results show that our proposed SA-GCN outperforms strong baseline models.

Introduction: The Challenge of Long-Range Dependencies

Aspect-Level Sentiment Classification (ALSC) aims to identify the sentiment polarity of specific aspect terms within a sentence. While Graph Convolutional Networks (GCNs) leveraging syntactic dependency trees have become a standard approach, they suffer from two primary limitations: the distance gap and over-smoothing.

In many natural language sentences, the opinion word that determines the sentiment for an aspect is located several hops away in the dependency tree. For instance, in the sentence "I thought learning Mac OS would be hard but picked it up easily," the aspect "Mac OS" is syntactically distant from the opinion phrase "picked up."

Example of a dependency tree illustrating the long-distance relationship between an aspect term and its opinion words.

To capture such long-range dependencies, standard GCNs require multiple layers. However, increasing the depth of a GCN often leads to over-smoothing, where node representations become indistinguishable, degrading classification performance. Most GCN-based models are thus restricted to shallow architectures (typically two layers), leaving them unable to bridge the distance gap effectively.

Proposed Methodology: SA-GCN

To address these issues, we propose the Selective Attention Based Graph Convolutional Network (SA-GCN). This model combines the strengths of self-attention—which can model global dependencies—with the structural inductive bias of GCNs.

Architecture of the Selective Attention Based Graph Convolutional Network (SA-GCN).

Model Architecture

The SA-GCN architecture consists of several key components:

  1. BERT Encoder: We utilize a pre-trained BERT-base model to generate contextualized word embeddings for the input sentence and aspect terms.
  2. Syntactic GCN: An initial GCN layer processes the standard dependency tree to capture local syntactic information.
  3. SA-GCN Block: This core component utilizes a multi-head self-attention mechanism to compute a dense attention matrix. To mitigate noise and prevent over-smoothing, we implement a Top-$k$ Selection procedure. This step sparsifies the attention matrix by retaining only the $k$ most relevant edges for each word, effectively learning a task-specific graph structure.
  4. Joint Training with Opinion Extraction: The model is trained multi-taskly. Alongside sentiment classification, it performs opinion word extraction using a Conditional Random Field (CRF) layer. This auxiliary task provides a strong supervisory signal, guiding the attention mechanism to focus on relevant opinion words.

Experimental Setup and Results

We evaluated SA-GCN on four benchmark datasets from SemEval (14Lap, 14Rest, 15Rest, and 16Rest). The model consistently outperformed strong BERT-based baselines, achieving state-of-the-art results.

Key Findings

  • Sparsity is Crucial: Experiments showed that a fully connected attention graph introduces significant noise. The top-$k$ selection mechanism is vital for filtering out irrelevant context words and focusing on the most informative opinion terms.
  • Robustness to Sentence Length: Unlike standard GCNs, which see a performance drop as sentence length (and dependency path length) increases, SA-GCN maintains stable performance. This confirms its ability to handle long-range dependencies without requiring excessive layers.
  • Multi-task Synergy: The inclusion of the opinion extraction task significantly boosts sentiment classification accuracy, demonstrating that identifying opinion words is a critical sub-step for accurate sentiment analysis.

Conclusion

The SA-GCN model effectively bridges the gap between local syntax and global context in aspect-level sentiment classification. By dynamically constructing a sparse, task-relevant graph through selective attention, it overcomes the limitations of fixed dependency trees and the over-smoothing problems of deep GCNs. Future work will explore more advanced graph neural network architectures to further enhance the model's depth and representational power.

Summary updated

Abstract

Recent work on aspect-level sentiment classification has employed Graph Convolutional Networks (GCN) over dependency trees to learn interactions between aspect terms and opinion words. In some cases, the corresponding opinion words for an aspect term cannot be reached within two hops on dependency trees, which requires more GCN layers to model. However, GCNs often achieve the best performance with two layers, and deeper GCNs do not bring any additional gain. Therefore, we design a novel selective attention based GCN model. On one hand, the proposed model enables the direct interaction between aspect terms and context words via the self-attention operation without the distance limitation on dependency trees. On the other hand, a top-k selection procedure is designed to locate opinion words by selecting k context words with the highest attention scores. We conduct experiments on several commonly used benchmark datasets and the results show that our proposed SA-GCN outperforms strong baseline models.
BibTeX
@misc{hou2021selective,
  author = {Xiaochen Hou and Jing Huang and Guangtao Wang and Peng Qi and Xiaodong He and Bowen Zhou},
  title = {{Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification}},
  year = {2021},
  doi = {10.18653/v1/2021.textgraphs-1.8},
}
Figure