A context-based model for Sentiment Analysis in Twitter

Andrea Vanzo, Danilo Croce, Roberto BasiliCOLING 2014

TL;DR

This paper introduces a novel context-aware model for Twitter sentiment analysis that leverages conversational threads and topical hashtags to improve accuracy. By treating sentiment analysis as a sequential classification problem and employing a Markovian SVM (SVMhmm), the authors demonstrate significant gains in detecting sentiment, especially for ambiguous tweets.

What Problem Does This Paper Address?

Sentiment analysis on social media, particularly Twitter, is challenging due to the informal language, misspellings, slang, and the brevity of individual tweets. Traditional sentiment analysis often treats each tweet in isolation, missing crucial contextual information. This paper addresses the limitation of context-free analysis by proposing methods to incorporate the surrounding conversation and topical discussions (identified by hashtags) into the sentiment detection process. The authors highlight that understanding the flow of a conversation or the broader topic of discussion can disambiguate the sentiment of individual, potentially ambiguous, tweets.

Methods

The core of the paper's contribution lies in its context-aware modeling approach. Instead of analyzing tweets individually, they are treated as sequences within a larger context.

  • Sequential Classification: The problem is reframed as a sequential classification task, where the sentiment of a target tweet is predicted based on a sequence of preceding tweets.
  • Markovian SVM (SVMhmm): To handle this sequential nature, the authors employ a Hidden Markov Model (HMM) extension of Support Vector Machines (SVMhmm). This allows the model to learn patterns and dependencies across sequences of tweets, not just individual ones.
  • Contextual Representations: They propose several ways to represent context:
  • Conversational Context: This involves building a sequence of tweets that are replies to each other, up to a certain length (l).
  • Topical Context: This leverages hashtags to aggregate tweets related to a specific topic. The context is formed by the l most recent tweets sharing at least one hashtag with the target tweet.
  • User Sentiment Profile Context (USPK): This approach aims to capture a user's general sentiment tendencies by analyzing their past tweets.
  • Kernel Functions: To integrate these different contextual features, they utilize various kernel functions:
  • Bag of Word Kernel (BoWK): A standard lexical overlap measure.
  • Lexical Semantic Kernel (LSK): Uses latent semantic analysis on word co-occurrence vectors for richer lexical representation.
  • User Sentiment Profile Kernel (USPK): Captures user-specific sentiment biases.
  • Multiple Kernel Learning: The paper explores combining these kernels (e.g., BoWK + LSK + USPK) to leverage multiple sources of information.

Datasets

The experiments were conducted using a subset of the ACL SemEval-2013 "Sentiment Analysis in Twitter" dataset. Specifically, the Training and Development portions were used to retrieve tweet identifiers, allowing for the retrieval of actual tweet contexts from Twitter servers. The data was split into Training, Held-out, and Test sets. The paper also categorizes the data into "whole dataset," "conversation-filtered," and "hashtag-filtered" subsets to evaluate performance under different contextual availability scenarios.

Key Insights and Ideas

  • Context is King (Especially for Ambiguity): The most significant insight is that contextual information (conversations and topics) dramatically improves sentiment analysis accuracy, particularly for tweets that are otherwise ambiguous when analyzed in isolation.
  • Sequential Modeling Pays Off: Treating tweets as sequences rather than independent entities unlocks the power of context, leading to substantial accuracy gains.
  • Different Contexts, Different Strengths: Conversational and topical contexts offer distinct but complementary information, and their combination can be powerful.
  • User Profiling's Nuance: While user sentiment profiles (USPK) offer a form of context, the paper suggests they might represent a more static view and can sometimes clash with dynamic conversational or topical contexts, leading to suboptimal results when combined without careful tuning.
  • Flexibility and Automation: The proposed approach is flexible and, importantly, does not require manually coded resources, making it adaptable and scalable.

Key Results

The paper reports significant improvements in sentiment detection accuracy using their context-aware models.

  • Overall Improvement: The sequential tagging approach, especially when combined with contextual information, achieved a relative increment in detection accuracy of around 20% in F1 measure on average.
  • Contextual Benefits: Context-driven models (SVMhmm) generally outperformed context-free multi-class SVMs.
  • Specific Gains:
    • For the whole dataset, context-free models with richer representations (LSK, USPK) showed improvements of up to +10.73%.
  • When using rich contexts (conversation-filtered and hashtag-filtered datasets), the SVMhmm approach showed striking benefits. For instance, the BoW Kernel over the conversation corpus saw a relative improvement of 18.26% in F1. For topical contexts, the BoW Kernel improved by 23.73%.
  • Optimal Context Size: The experiments suggest an optimal context size of around 6 tweets for conversational contexts and 16 tweets for topical contexts.

Limitations and Future Work

  • User Sentiment Profile Kernel (USPK) Effectiveness: The authors note that the USPK model was not very effective and seemed to clash with other contextual notions. They suggest that USPK might represent a more static context and that further investigation into optimal kernel combinations and history sizes is needed.
  • User Interaction Dynamics: The paper acknowledges that user interaction dynamics (reputation, authority, influence) are complex and deserve better representations for future exploration.
  • Optimal Kernel Combination: Learning the optimal combination of kernels remains an area for deeper investigation.
  • Larger Datasets: The authors believe that larger training datasets could lead to even better results, especially given the observed benefits of contextual information.

Summary updated

BibTeX
@inproceedings{vanzo2014context,
  author = {Andrea Vanzo and Danilo Croce and Roberto Basili},
  title = {{A context-based model for Sentiment Analysis in Twitter}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2014},
}
Figure