Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification

Xiaochen Hou, Jing Huang, Guangtao Wang, Kevin Huang, Xiaodong He, Bowen ZhouOpenReview 2019 · DOI · OpenReview

TL;DR

This paper introduces SA-GCN, a novel approach for Aspect-Level Sentiment Classification (ALSC) that leverages both syntactic information from dependency trees and a selective attention mechanism. By intelligently focusing on the most relevant context words, even those distant in the dependency parse, SA-GCN achieves state-of-the-art results on several benchmark datasets.

What Problem Does This Paper Address?

Aspect-Level Sentiment Classification (ALSC) is a fine-grained task that aims to determine the sentiment (positive, negative, or neutral) expressed towards a specific "aspect term" within a sentence. For example, in "Despite a slightly limited menu, everything prepared is done to perfection," the aspect term "menu" has a negative sentiment, indicated by "limited," while "everything prepared" has a positive sentiment due to "done to perfection."

While recent methods have successfully used Graph Convolutional Networks (GCNs) on dependency trees to capture syntactic relationships, they often struggle when the crucial opinion words are several hops away from the aspect term in the dependency parse. Standard GCNs, particularly those with more than two layers, suffer from over-smoothing, limiting their ability to capture these long-range dependencies. This paper addresses the challenge of effectively incorporating distant but important contextual information for accurate ALSC.

Methods

The proposed SA-GCN model builds upon existing GCN approaches for ALSC but introduces a novel "Selective Attention" block. Here's a breakdown:

  • BERT Encoder: The process begins by using a pre-trained BERT model to generate initial word embeddings for the sentence and aspect term. This captures rich semantic information.
  • GCN on Dependency Tree: These BERT embeddings are then used as node features in a GCN operating on the sentence's dependency tree. This step injects syntactic structure, allowing information to propagate between syntactically related words. However, as noted, this is typically limited to short distances.
  • Selective Attention Block (SA-GCN): This is the core innovation. For each word, a multi-head self-attention mechanism is applied to compute attention scores with all other words. Crucially, instead of using all these connections, a "top-k selection" strategy is employed. This mechanism identifies the k most important context words for each word, effectively creating a sparser graph. This allows the model to directly focus on potentially distant but highly relevant opinion words.
  • Head-Independent vs. Head-Dependent Selection: The paper explores two strategies for selecting these top-k words: head-independent (aggregating decisions across all attention heads) and head-dependent (each head makes its own selection). Head-independent selection is found to be more effective.
  • Second GCN Layer: A second GCN layer is applied to this new, sparser graph generated by the selective attention. This ensures that the selected top-k words are directly reachable by the aspect term, facilitating the learning of their interactions.
  • Classifier: Finally, the representation of the aspect term node is fed into a classifier (a two-layer MLP) to predict the sentiment polarity.
  • Auxiliary Opinion Extraction Task: To further enhance performance and provide interpretability, the model is jointly trained with an opinion extraction task. This task aims to identify words that express sentiment towards the aspect term, providing additional supervision for the top-k selection mechanism.

Datasets

The paper evaluates SA-GCN on four standard benchmark datasets for aspect-level sentiment classification:

  • SemEval 2014 Task 4 (Laptop reviews - 14Lap)
  • SemEval 2014 Task 4 (Restaurant reviews - 14Rest)
  • SemEval 2015 Task 12 (Restaurant reviews - 15Rest)
  • SemEval 2016 Task 5 (Restaurant reviews - 16Rest)

The paper does not introduce new datasets.

Key Insights and Ideas

  • Beyond Two-Hop Reachability: The core insight is that relying solely on short-range syntactic information (like that captured by 2-layer GCNs) is insufficient when opinion words are far from the aspect term. The proposed selective attention mechanism directly addresses this limitation.
  • The Power of Sparsity: By selectively focusing on the top-k most relevant context words, the model avoids the "noise" and over-smoothing issues that can arise from dense graph connections, leading to more focused and effective feature aggregation.
  • Joint Training for Synergy: The auxiliary opinion extraction task is not just for interpretability; it actively helps the sentiment classification task by providing better supervision for the selective attention mechanism, guiding it to identify truly opinion-bearing words.
  • Head-Independent Attention is Key: The finding that head-independent selection outperforms head-dependent selection suggests that a consensus across multiple attention perspectives is more robust than individual head decisions for identifying crucial context.

Key Results

  • State-of-the-Art Performance: SA-GCN achieves new state-of-the-art results on three out of the four benchmark datasets (15Rest, 16Rest, and 14Lap) in terms of accuracy and Macro-F1. It also achieves competitive results on the 14Rest dataset.
  • Improved Accuracy with Joint Training: Jointly training SA-GCN with the opinion extraction task consistently boosts performance across all datasets.
  • Effectiveness on Long-Distance Dependencies: Qualitative analysis and sentence length analysis demonstrate that SA-GCN significantly outperforms a standard 2-layer GCN model, especially on longer sentences where long-distance dependencies are more prevalent.
  • Opinion Extraction Benefits: The joint SA-GCN model also shows strong performance on the opinion extraction task, outperforming existing baselines like IOG and ASTE.

Limitations and Future Work

  • Limited Number of SA-GCN Blocks: The study found that stacking multiple SA-GCN blocks did not improve performance and could even lead to overfitting, likely due to the limited size of the training datasets. The authors suggest exploring deeper GNN models suitable for this task in future work.
  • Hyperparameter Sensitivity: The performance is sensitive to the choice of k (number of context words) and the number of SA-GCN blocks. The optimal k was found to be small (around 3), and a single block was often sufficient.
  • Future Work: The authors plan to investigate deeper GNN models and explore more sophisticated ways to integrate syntactic and semantic information. They also mention releasing their code.

Summary updated

Abstract

Aspect-level sentiment classification aims to identify the sentiment polarity towards a specific aspect term in a sentence. Most current approaches mainly consider the semantic information by utilizing attention mechanisms to capture the interactions between the context and the aspect term. In this paper, we propose to employ graph convolutional networks (GCNs) on the dependency tree to learn syntax-aware representations of aspect terms. GCNs often show the best performance with two layers, and deeper GCNs do not bring additional gain due to over-smoothing problem. However, in some cases, important context words cannot be reached within two hops on the dependency tree. Therefore we design a selective attention based GCN block (SA-GCN) to find the most important context words, and directly aggregate these information into the aspect-term representation. We conduct experiments on the SemEval 2014 Task 4 datasets. Our experimental results show that our model outperforms the current state-of-the-art.
BibTeX
@misc{hou2019selective,
  author = {Xiaochen Hou and Jing Huang and Guangtao Wang and Kevin Huang and Xiaodong He and Bowen Zhou},
  title = {{Selective Attention Based Graph Convolutional Networks for Aspect-Level Sentiment Classification}},
  year = {2019},
  eprint = {1910.10857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.10857},
  howpublished = {OpenReview},
}
Figure