Leveraging Multi-grained Sentiment Lexicon Information for Neural Sequence Models

Yan Zeng, Yangyang Lan, Yazhou Hao, Chen Li, Qinhua Zheng

TL;DR

This paper introduces a novel method to enhance neural sentiment analysis models by incorporating multi-grained lexicon information. By enriching word embeddings with sentiment properties and guiding attention mechanisms with coarse-grained lexicon labels, the proposed approach significantly improves classification accuracy on standard benchmarks, even rivaling models that rely on expensive phrase-level annotations.

What Problem Does This Paper Address?

Sentiment analysis, particularly at the sentence level, has seen great strides with neural network models. However, many of these models are either overly complex or require costly feature engineering. While existing linguistic resources like sentiment lexicons, negation words, and intensifiers hold valuable information, current methods often underutilize them. A common approach is to simply embed positive/negative labels, neglecting the nuanced impact of negation and intensification. This paper aims to bridge this gap by proposing a more effective and general way to integrate these linguistic resources into neural sequence models, making them more accurate and less reliant on complex architectures or expensive annotations.

Methods

The core of this paper's contribution lies in its novel approach to incorporating lexicon information through two distinct granularities:

  1. Fine-grained Sentiment Embedding: Words are annotated with five categories: "not in" lexicon, positive, negative, negation words, and intensifiers. These fine-grained labels are then used to learn sentiment property embeddings. These learned embeddings are concatenated with pre-trained word embeddings (like GloVe), effectively injecting explicit sentiment "properties" into each word's representation.
  2. Coarse-grained Enhanced Attention Mechanism: For the attention mechanism, a simpler, coarse-grained annotation is used: words are either "in" a lexicon or "not in" one. This information is used to guide the attention mechanism, encouraging it to assign higher weights to words identified as sentiment-related. This helps the model focus more on the most informative parts of a sentence for sentiment classification.

The overall architecture presented is a standard Bi-LSTM model enhanced with these lexicon-based features.

Datasets

The paper evaluates its method on two standard sentiment analysis datasets:

  • Stanford Sentiment Treebank (SST): This dataset provides sentence-level annotations with five classes (very negative, negative, neutral, positive, very positive). Notably, SST also offers phrase-level annotations, but the authors explicitly choose not to use them to demonstrate their method's ability to avoid expensive annotations.
  • Movie Review (MR): This dataset is used for binary sentiment classification (positive/negative).

The authors also mention collecting their own sentiment lexicon, comprising 2759 positive words, 5111 negative words, 35 negation words, and 62 intensifiers, which they have released.

Key Insights and Ideas

  • Lexicons are more than just positive/negative: The paper highlights that negation words and intensifiers play a crucial role in sentiment and are often overlooked by simpler embedding strategies.
  • Multi-granularity is key: Using different levels of detail for different model components (fine-grained for embeddings, coarse-grained for attention) proves effective. This suggests a flexible way to leverage linguistic knowledge.
  • Soft constraints are robust: The method's ability to handle potentially incorrect lexicon annotations is a significant advantage. By incorporating lexicon information as soft constraints (through embeddings and attention weights) rather than hard rules, the model can still perform well even if some annotations are wrong.
  • Attention can be guided by external knowledge: The paper demonstrates that external linguistic knowledge can be effectively used to enhance attention mechanisms, making them more discriminative for the task.

Key Results

  • The proposed enhanced Bi-LSTM model achieved state-of-the-art results on both the SST-5 and MR datasets.
  • Specifically, the enhanced Bi-LSTM model was able to compare favorably with a Tree-LSTM model, which typically relies on expensive phrase-level annotations. This underscores the value of the lexicon-based approach in achieving high performance without costly data annotation.
  • The paper reports that their model failed to capture extra information in only 3.58% of test cases, indicating the general effectiveness of the lexicon integration.

Limitations and Future Work

  • Lexicon size and quality: The authors acknowledge that the performance of their method is dependent on the quality and size of the lexicon. They suggest that improving these aspects could lead to further significant gains in classification accuracy.
  • Attention mechanism performance: While the attention mechanism was enhanced, the authors note that in most cases, the attention scores showed only slight differences, and common words sometimes received higher scores than expected. They suggest that further refinement of the attention component might be beneficial.
  • Reusability of lexicon resource: The authors express optimism that their released lexicon resource can be reused for future research in sentiment analysis.

Summary updated

Abstract

Neural sequence models have achieved great success in sentence-level sentiment classification. However, some models are exceptionally complex or based on expensive features. Some other models recognize the value of existed linguistic resource but utilize it insufficiently. This paper proposes a novel and general method to incorporate lexicon information, including sentiment lexicons(+/-), negation words and intensifiers. Words are annotated in fine-grained and coarse-grained labels. The proposed method first encodes the fine-grained labels into sentiment embedding and concatenates it with word embedding. Second, the coarse-grained labels are utilized to enhance the attention mechanism to give large weight on sentiment-related words. Experimental results show that our method can increase classification accuracy for neural sequence models on both SST-5 and MR dataset. Specifically, the enhanced Bi-LSTM model can even compare with a Tree-LSTM which uses expensive phrase-level annotations. Further analysis shows that in most cases the lexicon resource can offer the right annotations. Besides, the proposed method is capable of overcoming the effect from inevitably wrong annotations.
BibTeX
@misc{zeng2018leveraging,
  author = {Yan Zeng and Yangyang Lan and Yazhou Hao and Chen Li and Qinhua Zheng},
  title = {{Leveraging Multi-grained Sentiment Lexicon Information for Neural Sequence Models}},
  year = {2018},
  eprint = {1812.01527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.01527},
}

Loading related content...

Figure