Tweester at SemEval-2016 Task 4: Sentiment Analysis in Twitter Using Semantic-Affective Model Adaptation

Elisavet Palogiannidi, Athanasia Kolovou, Fenia Christopoulou, Filippos Kokkinos, Elias Iosif, Nikolaos Malandrakis, Haris Papageorgiou, Shrikanth S. Narayanan, Alexandros PotamianosSemEval @ NAACL 2016 · DOI · OpenReview

TL;DR

This paper presents a novel system for sentiment analysis in Twitter, combining multiple independent models (neural networks, semantic-affective models, topic modeling) with a probabilistic fusion scheme. Their key innovation is adapting the semantic-affective space for each tweet using topic modeling, leading to a first-place finish in the SemEval-2016 Task 4, Subtask B.

What Problem Does This Paper Address?

The paper tackles the challenge of accurately determining sentiment (positive, negative, neutral) in tweets. Twitter's informal language, diverse topics, and unconventional grammar make this a difficult task. Existing approaches often rely on bag-of-words models with Twitter-specific features, word embeddings, or deep learning. However, adapting sentiment analysis models to the specific semantic context of individual tweets remains an open area for improvement.

Methods

The authors developed a multi-system approach for sentiment analysis, combining several classifiers and using a late fusion scheme for the final decision.

  • Baseline System: An enhanced version of a previous SemEval submission, incorporating affective features derived from semantic similarity and a topic modeling approach.
  • Semantic-Affective Model: This model assumes semantic similarity implies affective similarity. It builds a semantic model and then estimates affective ratings for unknown tokens by leveraging similar words.
  • Topic Modeling Adaptation: A core novelty is using topic modeling (specifically LDA) to adapt the semantic-affective space for each tweet. The corpus is split based on detected topics, and semantic models are built for each subcorpus, allowing for context-specific sentiment analysis.
  • Neural Network (CNN): A deep convolutional neural network inspired by sentence classification tasks was developed. Tweets are represented as matrices of word embeddings, and the CNN learns features through convolution and max-pooling operations.
  • Word2Vec System: This system exclusively relies on semantic representations from word embeddings. Tweet-level vectors are generated by averaging word vectors, and a random forest classifier is trained on these.
  • Webis System: An ensemble of existing systems that performed well in previous SemEval tasks.
  • Fusion Scheme: A late fusion scheme is employed, where the posterior probabilities from individual classifiers are combined to make the final decision. A mean algebraic fusion scheme was selected for the reported results.

Datasets

  • General Purpose Corpus (G-116M): A large corpus of 116 million sentences collected from web search engine snippets.
  • Twitter Specific Dataset (T-115M): A dataset of 115 million tweets.
  • SemEval 2016 Datasets: Training data for SemEval-2016 Task 4, Subtasks A and B.
  • SemEval 2013 Dataset: Training data from SemEval-2013.
  • ANEW (Affective Norms for English Words): Used for bootstrapping the affective lexicon expansion process.
  • Google n-gram corpus and Twitter n-gram corpus: Used for hashtag expansion.

Key Insights and Ideas

  • Topic Modeling for Contextual Adaptation: The most significant insight is that adapting the semantic-affective space to the specific topic of a tweet using topic modeling can substantially improve sentiment analysis performance. This moves beyond generic sentiment lexicons to more context-aware analysis.
  • Ensemble of Diverse Models: Combining diverse modeling approaches (neural networks, semantic models, topic models) is crucial for capturing different aspects of sentiment and achieving robust performance.
  • Late Fusion Effectiveness: A late fusion scheme, combining the outputs of multiple classifiers, proved effective in aggregating the strengths of individual systems.
  • Importance of Feature Engineering: While deep learning is powerful, careful feature engineering, including affective features, word embeddings, and handling of negations and multiword expressions, still plays a vital role.

Key Results

  • Subtask B (Tweet Classification): The authors' submitted system ranked first place in SemEval-2016 Task 4, Subtask B.
  • Subtask A (Message Polarity Classification): The submitted system ranked fifth place.
  • System Combinations: Experimenting with various system combinations for Subtask B showed that performance could be improved beyond the submitted system, with some combinations achieving higher macro-averaged recall and F1 scores. For instance, combining the baseline with other systems yielded robust performance.

Limitations and Future Work

  • Subtask A Performance: While the system performed well in Subtask B, its ranking in Subtask A was lower. Future work could focus on improving performance for the three-point sentiment classification.
  • Fusion Enhancement: The authors suggest that further exploration of fusion schemes and system combinations could lead to even higher performance.
  • Contextual Adaptation: While topic modeling was used for adaptation, further research into more sophisticated methods for capturing tweet-specific context could be beneficial.

Summary updated

BibTeX
@inproceedings{palogiannidi2016tweester,
  author = {Elisavet Palogiannidi and Athanasia Kolovou and Fenia Christopoulou and Filippos Kokkinos and Elias Iosif and Nikolaos Malandrakis and Haris Papageorgiou and Shrikanth S. Narayanan and Alexandros Potamianos},
  title = {{Tweester at SemEval-2016 Task 4: Sentiment Analysis in Twitter Using Semantic-Affective Model Adaptation}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1023},
}
Figure