Language Independent Sentiment Analysis with Sentiment-Specific Word Embeddings

Carl Saroufim, Akram Almatarky, Mohammad Abdel HadyEMNLP 2018

TL;DR

This paper introduces Sentiment-Specific Word Embeddings (SSWE) to improve sentiment analysis, especially for languages with limited labeled data. By fine-tuning pre-trained Word2Vec embeddings on auto-labeled tweets using emojis, they achieve better sentiment classification performance than standard Word2Vec, particularly when using deep learning models.

What Problem Does This Paper Address?

Training effective sentiment analysis models typically requires large amounts of manually labeled data, which is expensive and time-consuming to create, especially for languages other than English. While word embeddings like Word2Vec capture semantic and syntactic information, they often fail to adequately distinguish between words with opposite polarities (e.g., "good" and "bad" might be close in embedding space). This paper aims to bridge this gap by developing a language-independent method to create sentiment-aware word embeddings that improve sentiment classification performance, even with limited labeled data.

Methods

The core of this work is the creation of Sentiment-Specific Word Embeddings (SSWE). The process involves:

  • Language-Independent Auto-Labeling: Leveraging the universality of emojis and emoticons, the authors auto-label a large corpus of French tweets. Tweets containing specific positive or negative emojis are assigned the corresponding sentiment label. This bypasses the need for manual annotation or language-specific lexicons.
  • Unsupervised Word2Vec Training: A standard Word2Vec model is trained on the auto-labeled tweets to capture general word representations.
  • SSWE Training: The pre-trained Word2Vec embeddings are then used to initialize an embedding layer in a deep learning model (either a Recurrent Neural Network - RNN, or a Convolutional Neural Network - CNN). This deep learning model is trained on the auto-labeled data. Through backpropagation, the word embeddings are updated to incorporate sentiment information, creating the SSWE.
  • Transfer Learning: The trained SSWE are then used as features for a sentiment classification task on new datasets. The authors also explore transfer learning by fixing the SSWE and fine-tuning the subsequent layers of the deep learning model.
  • Preprocessing Variations: Two preprocessing steps were explored:
  • Preprocessing A: Removing emojis from the auto-labeled data before training SSWE to prevent the model from relying too heavily on emoji signals.
  • Preprocessing B: Replacing sequences of three or more identical characters with two to handle informal language and reduce feature dimensionality.

Datasets

  • Auto-Labeled French Tweets: A large corpus of French tweets was collected and auto-labeled using positive and negative emojis. Statistics on the size and polarity distribution are provided.
  • SemEval 2016 Competition Datasets:
    • Restaurant Reviews (Train): Used for training the sentiment classification models.
  • Restaurant Reviews (Test1): Used for evaluating models on data from the same domain as the training set.
  • Museum Reviews (Test2): Used for evaluating model generalization to an out-of-domain dataset.
  • Manually Labeled French Tweets (Test3): A dataset of French tweets without emojis, used to evaluate performance on real-world, clean data.

Key Insights and Ideas

  • Emojis as a Universal Labeling Tool: Emojis provide a surprisingly effective and language-independent way to generate large-scale sentiment-labeled datasets, overcoming the scarcity of manually labeled data in many languages.
  • SSWE Captures Sentiment Better: Sentiment-specific word embeddings (SSWE) demonstrably improve sentiment classification performance compared to standard Word2Vec embeddings, especially when the SSWE are trained on data where emojis are removed to avoid bias.
  • Deep Learning and Transfer Learning Synergy: Combining SSWE with deep learning architectures (RNN/CNN) and employing transfer learning yields the best results, indicating that fine-tuning pre-trained sentiment-aware embeddings is highly effective.
  • Emoji Bias is Real: Training SSWE on auto-labeled data that includes emojis can negatively impact performance on datasets with few or no emojis, highlighting the importance of cleaning the auto-labeled data for broader applicability.
  • Character Repetition Nuance: The paper suggests that character repetition in informal text might carry sentiment emphasis, and removing it might not always be beneficial.

Key Results

  • Logistic Regression: SSWE (specifically SSWE_C_no_emoji and SSWE_R_last_no_emoji) combined with ngrams achieved the best results among traditional models, showing improvements of up to 2.37% AUC on Test2 compared to the best baseline (ngram + Word2Vec).
  • Deep Learning (Transfer Learning): The most significant improvements were achieved using transfer learning with RNNs and SSWE trained without emojis. This approach yielded AUC improvements of 2.6%, 7.72%, and 3.88% on Test1, Test2, and Test3, respectively, over the best baseline.
  • SSWE vs. Word2Vec: SSWE consistently outperformed Word2Vec when used as features for sentiment classification, particularly on out-of-domain and cleaner datasets.

Limitations and Future Work

  • Emoji Dependency: The auto-labeling method relies on the presence of emojis. While effective, it might not capture sentiment in texts entirely devoid of them.
  • Language-Specific Features Ignored: The method is language-independent by design, meaning it doesn't leverage linguistic features like stemming, negation handling, or language-specific lexicons, which could potentially further boost performance.
  • Neutral Class Exclusion: The current work focuses on binary (positive/negative) classification and excludes neutral sentiment.
  • Future Work Suggestions:
    • Explore graph propagation techniques with different embedding models.
  • Investigate training SSWE on more general datasets (e.g., movie reviews) instead of just tweets.
  • Address the neutral sentiment class by introducing thresholds or training a separate subjectivity classifier.
  • Incorporate sentiment lexicons, stemming, normalization, and negation handling.

Summary updated

BibTeX
@inproceedings{saroufim2018language,
  author = {Carl Saroufim and Akram Almatarky and Mohammad Abdel Hady},
  title = {{Language Independent Sentiment Analysis with Sentiment-Specific Word Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure