Refined Global Word Embeddings Based on Sentiment Concept for Sentiment Analysis

Yabing Wang, Guimin Huang, Jun Li, Hui Li, Ya Zhou, Hua JiangIEEE Access 2021 · DOI

Introduction

Sentiment analysis remains a cornerstone of natural language processing (NLP), with applications spanning political forecasting, news monitoring, and market research. At its heart, the effectiveness of sentiment analysis depends heavily on word embeddings—vector representations that capture the semantic meaning of words. However, traditional embeddings often fail to capture the nuanced emotional shifts that occur when a word is used in different contexts. This paper introduces Refined Global Word Embeddings (RGWE), a novel framework that leverages "sentiment concepts" and a fused intensity lexicon to provide more accurate, context-aware sentiment representations.

The Challenges of Traditional Embeddings

Existing word embedding models like Word2Vec and GloVe primarily rely on distributional semantics—the idea that words appearing in similar contexts have similar meanings. While powerful, this approach faces two significant hurdles in sentiment analysis:

  1. The Similarity Trap: Words with opposite polarities, such as "excellent" and "terrible," often appear in identical syntactic structures (e.g., "The service was ___"). Consequently, traditional models may place these antonyms close together in the vector space, making it difficult for classifiers to distinguish between positive and negative sentiments.
  2. Context-Dependent Polysemy: Many words carry sentiment only in specific contexts. For instance, the word "blue" is neutral when describing a color ("a blue car") but negative when describing an emotion ("feeling blue"). Static sentiment embeddings typically assign a single, fixed score to a word, failing to capture these situational shifts.

Methodology: The RGWE Framework

To address these issues, the authors propose a multi-layered refinement process that integrates external knowledge with distributional features.

1. Fusion Sentiment Intensity Lexicon (FSIL)

Rather than relying on simple binary labels (positive/negative), the researchers constructed the Fusion Sentiment Intensity Lexicon (FSIL). By integrating six major lexicons—including AFINN, SentiWordNet, and SenticNet—they created a comprehensive resource of over 172,000 words. This lexicon provides granular intensity scores, allowing the model to recognize that "ecstatic" represents a stronger positive sentiment than "happy."

2. Sentiment Concept Disambiguation

The core innovation of this paper is the use of the Microsoft Concept Graph to disambiguate word meanings. For any given word in a sentence, the model identifies its "optimal sentiment concept" based on the surrounding context. For example, if the word "pink" appears in a sentence about fashion, its concept is identified as "color" (neutral). If it appears in a context referring to high-ranking officials, its concept might be "elite" (positive). By anchoring sentiment to these concepts, the model avoids mislabeling neutral words.

3. Dual Refinement Strategy

The RGWE model combines two refined embedding techniques:

  • Refined-Word2Vec: This component uses an iterative optimization algorithm to adjust word vectors. It pulls a word's vector closer to its "sentiment neighbors"—words that share both a similar concept and a similar sentiment intensity.
  • Refined-GloVe: This model incorporates sentiment weights derived from TF-IDF and the FSIL. Crucially, it introduces position encoding to capture how the proximity of a word to sentiment-bearing terms affects the overall meaning of a sentence.
Position encoding mechanism for capturing contextual sentiment influence.

The final RGWE representation is an ensemble of these two models, providing a balanced view of internal semantic relationships and external sentiment knowledge.

Experimental Evaluation

The authors evaluated RGWE across six diverse datasets, including Twitter sentiment (SemEval-2013), movie reviews (IMDB, SST1, SST2), and consumer reviews (Amazon, Yelp 2014). These datasets cover both binary and fine-grained (5-class) sentiment classification tasks.

Key Results

Using deep learning architectures like Bi-GRU and Bi-LSTM, the RGWE embeddings demonstrated superior performance:

  • Accuracy Gains: On the IMDB dataset, RGWE achieved an F1-score of 91.3% with a Bi-GRU, compared to 84.5% for standard Word2Vec.
  • Fine-Grained Precision: The model excelled in tasks requiring the distinction between subtle emotional levels (e.g., "somewhat positive" vs. "very positive"), validating the effectiveness of the intensity-based FSIL and the concept-based disambiguation.

Conclusion and Future Work

The RGWE framework represents a significant step forward in making word embeddings context-aware for sentiment analysis. By moving beyond static labels and incorporating conceptual knowledge, the model achieves state-of-the-art results on multiple benchmarks.

A current limitation is that the Microsoft Concept Graph primarily focuses on nouns, which restricts the "Sentiment Concept" refinement for verbs and adjectives. Future research will aim to expand this conceptual mapping to all parts of speech, creating a truly universal sentiment-aware embedding space.

Summary updated

Abstract

Sentiment Analysis is an important research direction of natural language processing, and it is widely used in politics, news and other fields. Word embeddings play a significant role in sentiment analysis. The existing sentiment embeddings methods directly embed the sentiment lexicons into traditional word representation. This sentiment representation methods can only differentiate the sentiment information of different words, not the same word in different contexts, so it cannot provide accurate sentiment information for word in different contexts. This paper proposes sentiment concept to solve the problem. First, we found the optimal sentiment concept of words in Microsoft Concept Graph according to the context of words. Then we obtained the sentiment information of words under optimal sentiment concept from the multi-semantics sentiment intensity lexicon which we constructed in this paper to achieve accurate embedding of sentiment information and provide more accurate semantics and sentiment representation for words. Finally, we combined two refined word embeddings methods to achieve a more comprehensive word representation. Compared with traditional and sentiment embeddings methods on six representative datasets, the validity of the word embeddings method based on sentiment concept proposed in this paper is verified.
BibTeX
@inproceedings{wang2021refined,
  author = {Yabing Wang and Guimin Huang and Jun Li and Hui Li and Ya Zhou and Hua Jiang},
  title = {{Refined Global Word Embeddings Based on Sentiment Concept for Sentiment Analysis}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3062654},
}
Figure