Ternary Twitter Sentiment Classification with Distant Supervision and Sentiment-Specific Word Embeddings

Mats Byrkjeland, Frederik Gørvell de Lichtenberg, Björn GambäckEMNLP 2018

TL;DR

This paper introduces the Ternary Sentiment Embedding Model, a novel approach to creating sentiment-specific word embeddings that accounts for positive, negative, and neutral sentiment. By leveraging distant supervision on a large Twitter corpus, the model demonstrates improved sentiment classification performance compared to existing methods, highlighting the critical role of dataset quality in training effective embeddings.

What Problem Does This Paper Address?

Traditional word embeddings capture semantic relationships but struggle to differentiate between words with opposite sentiments that appear in similar contexts (e.g., "good" vs. "bad"). Sentiment-specific word embeddings aim to address this by incorporating sentiment information. However, existing methods often focus on binary (positive/negative) sentiment and require large amounts of manually annotated data, which is expensive to obtain. This paper tackles the challenge of creating effective sentiment embeddings for a ternary sentiment classification task (positive, negative, neutral) by exploring and comparing different distant supervision methods for automatically labeling large datasets, thereby reducing reliance on manual annotation.

Methods

The core contribution is the Ternary Sentiment Embedding Model, an extension of the Hybrid Ranking Model. This new model features:

  • Ternary Sentiment Classification: It explicitly models three sentiment classes (positive, negative, neutral), unlike previous binary approaches.
  • Dual-Layer Architecture: It uses two parallel top layers:
  • A Context Linear Layer (similar to previous work) that learns word embeddings based on context.
  • A new Ternary Sentiment Linear Layer that learns sentiment scores for each of the three classes.
  • Weighted Loss Function: The total loss combines the context loss and the sentiment loss, allowing for a tunable balance between them.
  • Distant Supervision: The model is trained on weakly labeled data automatically generated from tweets using various methods, including emoticons, emojis, and lexicon-based classifiers.

The paper also compares several distant supervision methods for generating these weak labels, evaluating their effectiveness in producing high-quality training data for sentiment embeddings.

Datasets

The paper utilizes several datasets:

  • SemEval Datasets (2013-2016): These are the primary manually annotated datasets used for evaluating the performance of the trained sentiment embeddings and the overall Twitter Sentiment Analysis system. The paper notes that some tweets from these datasets were deleted, resulting in a smaller effective dataset size.
  • Large Twitter Corpus (approx. 547 million tweets): This corpus is used for generating weakly labeled data through various distant supervision methods.
  • Newly Generated Ternary Datasets: The paper describes the process of creating ternary-labeled datasets using different distant supervision techniques (Emoticons, Emojis, AFINN, TextBlob, VADER, Lexicon Classifier, and combinations thereof) from the large Twitter corpus. These datasets are crucial for training the Ternary Sentiment Embedding Model.

Key Insights and Ideas

  • Ternary Sentiment is Crucial: Explicitly modeling neutral sentiment significantly improves performance compared to binary approaches, especially when using sophisticated distant supervision.
  • Distant Supervision Quality Matters: The choice of distant supervision method has a substantial impact on the quality of the resulting sentiment embeddings and the final classification performance. Lexicon-based methods and ensemble approaches generally yield better results.
  • Context vs. Sentiment Trade-off: The weighting parameter (alpha) in the loss function reveals that context is slightly more important than sentiment for this task, but sentiment information is still vital and cannot be omitted.
  • Tweet Characteristics: Shorter context windows (size 3) are found to be optimal for Twitter data, suggesting that longer excerpts might introduce too much noise for the informal language used.
  • Embedding Dimensionality: Higher embedding dimensions (up to 150) generally lead to better performance, but with diminishing returns and increased computational cost.

Key Results

  • The Ternary Sentiment Embedding Model outperforms the Hybrid Ranking Model (Tang et al., 2016) on most distant supervision datasets, particularly when using more sophisticated methods that capture ternary sentiment.
  • When integrated into a full Twitter Sentiment Analysis system (Ternary Sentiment Embedding Model + SVM), the proposed approach achieves competitive results, outperforming the Hybrid Ranking Model and other baselines.
  • The best performing distant supervision method for training the Ternary Sentiment Embedding Model was the Lexicon Classifier, achieving a Macro F1 score of 0.6383 when averaged over epochs.
  • The final TSA system achieved a Macro F1 score of 0.655 when tested on the combined SemEval 2013-2016 datasets, which is competitive but does not match the absolute top SemEval results, potentially due to data differences and optimization for Macro F1.

Limitations and Future Work

  • Data Deletion: A significant number of tweets from the SemEval datasets were deleted, limiting the available data for training and evaluation.
  • Polysemy and Word Senses: The current model assumes all senses of a word are synonyms and that all words in a tweet share the same sentiment. Future work could extend the model to handle polysemous words and word-sense disambiguation for ternary sentiment.
  • Prior Polarity: The model doesn't explicitly leverage prior sentiment polarity of words. Future work could incorporate word-sense aware lexica to address this.
  • Comparison to State-of-the-Art: While competitive, the system does not surpass the absolute top results from SemEval, possibly due to differences in training data size, features used, or optimization metrics.
  • Computational Cost: Training the Ternary Sentiment Embedding Model is computationally intensive, though word2vec models offer a faster alternative with slightly lower performance.

Summary updated

BibTeX
@inproceedings{byrkjeland2018ternary,
  author = {Mats Byrkjeland and Frederik Gørvell de Lichtenberg and Björn Gambäck},
  title = {{Ternary Twitter Sentiment Classification with Distant Supervision and Sentiment-Specific Word Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure