WarwickDCS: From Phrase-Based to Target-Specific Sentiment Recognition

Richard Townsend, Adam Tsakalidis, Yiwei Zhou, Bo Wang, Maria Liakata, Arkaitz Zubiaga, Alexandra I. Cristea, Rob ProcterSemEval @ NAACL 2015 · DOI

TL;DR

This paper presents a hybrid approach to sentiment analysis on Twitter, combining traditional lexical features with modern word embeddings and syntactic parsing. They achieve strong results across phrase-level, tweet-level, and target-specific sentiment tasks, demonstrating the power of integrating diverse feature sets and exploring novel ways to leverage word embeddings for sentiment.

What Problem Does This Paper Address?

Sentiment analysis on social media, particularly Twitter, is challenging due to the informal language, brevity, and evolving nature of online communication. While lexicon-based approaches have been effective, they can be time-consuming to develop and domain-dependent. The authors aim to improve sentiment recognition by exploring hybrid systems that combine traditional lexicons with more advanced techniques like word embeddings and syntactic parsing. Specifically, they address three subtasks: 1) phrase-level sentiment (identifying sentiment within a specific phrase), 2) tweet-level sentiment (overall sentiment of a tweet), and 3) target-specific sentiment (sentiment towards a particular entity or topic within a tweet). The latter is particularly challenging as it requires understanding context and relevance to a specific target.

Methods

The core of their approach is a hybrid system that leverages multiple feature types.

  • Phrase-Based Sentiment (Subtask A): They combine lexical features (from several established lexicons like Bing Liu's and NRC's Emotion Lexicon) with n-grams (unigrams and bigrams) and word embeddings. A novel aspect is their exploration of two strategies for learning phrase-level sentiment-specific word embeddings using word2vec, aiming to capture sentiment in the immediate context of a phrase.
  • Tweet-Level Sentiment (Subtask B): This subtask builds upon Subtask A by using a similar combination of hybrid features (lexical, n-grams, word embeddings) fed into an SVM classifier. They also incorporate word embeddings generated from "happy" and "non-happy" tweets, as well as those from Tang et al. (2014b), to represent the entire tweet.
  • Target-Specific Sentiment (Subtask C): This is the most complex task. They develop several strategies:
  • Syntactic Relations: They use dependency parsing (Stanford parser) to identify the "topic-relevant portion" of a tweet as the weakly connected components of the dependency graph around a target word.
  • Per-Token Annotations: They generate sentiment annotations for individual tokens using their Subtask A classifier and then aggregate these.
  • Dependency Graph Propagation: Systems like CONLL-PROPAGATION build a dependency graph, attach 1-WINDOW sentiment to nodes, and propagate these values to the target.
  • Subgraph Matching: The SUBMISSION system attempts to match directed co-dependency graphs against previously seen trees to capture syntactic features relevant to sentiment.
  • Baseline: A simple approach that considers tokens around the target, demonstrating that even simpler methods can be competitive.

Datasets

The paper primarily uses data from the SemEval 2015 competition for sentiment analysis.

  • Training Data: They mention using a training set of 7,643 tweets for Subtask A, and a combined dataset of 9,684 tweets for Subtask B. For word embeddings, they utilize tweets collected by Purver and Battersby (2012) containing emoticons and hashtags.
  • Test Data: They evaluate on official SemEval test sets ("OT" and "PT" for Subtask A, and test/test progress datasets for Subtask B). For Subtask C, they use the official SemEval test data.
  • Development Data: They use development sets for fine-tuning parameters and evaluating feature sets.

Key Insights and Ideas

  • Word Embeddings for Sentiment are Powerful: The paper strongly advocates for the use of word embeddings, especially sentiment-specific ones, showing they can achieve performance comparable to or even exceeding lexicon-based approaches.
  • Phrase-Level Embeddings Matter: Their novel approach to learning phrase-level sentiment-specific word embeddings is a key contribution, suggesting that capturing sentiment in localized contexts is crucial.
  • Hybrid Systems Win: Combining diverse feature sets (lexical, n-grams, embeddings, syntactic) consistently yields better results than relying on a single type.
  • Syntactic Information is Useful for Target-Specific Sentiment: Dependency parsing helps identify relevant parts of a tweet for target-specific sentiment, though its effectiveness can be limited by parsing accuracy on noisy Twitter data.
  • Simpler Baselines Can Be Competitive: The BASELINE system for Subtask C, despite its simplicity, performed surprisingly well, highlighting the importance of robust feature selection and the potential for over-complexity in some advanced methods.

Key Results

  • Subtask A: Their Subtask A classifier ranked fourth in the official SemEval results. Combinations of features, particularly lexica with embeddings, showed strong performance (e.g., 81.71 in OT and 84.09 in PT for embeddings + extra features).
  • Subtask B: They achieved ranks of 27 and 24 out of 40 teams on the test datasets for Subtask B, with an average F1 score of 65.78 on the development set. They found that their own learned sentiment-specific word embeddings contributed more than Tang's embeddings.
  • Subtask C: Their BASELINE and µPARSE classifiers for Subtask C would have ranked second if submitted. The SUBMISSION-SENTIMENT system achieved the highest score among their submitted systems (29.37), but BASELINE (46.59) and PARSE (46.87) performed significantly better.

Limitations and Future Work

  • Dependency Parsing on Twitter: The authors acknowledge that dependency parsing on Twitter data can be noisy and computationally expensive, which impacts the effectiveness of some Subtask C approaches. They suggest that further investigation is needed to better exploit the relationship between topic-specific and phrase-level sentiment.
  • Complexity vs. Performance: While advanced methods like dependency graph propagation were explored, simpler approaches like BASELINE often performed better, indicating a need for more robust and efficient methods for Twitter parsing.
  • Future Work: The paper suggests that exploring simpler approaches for dependency parsing on Twitter and further investigating how to exploit syntactic information in conjunction with phrase-level sentiment are promising directions. They also highlight the significant potential of word embeddings for future sentiment analysis research.

Summary updated

BibTeX
@inproceedings{townsend2015warwickdcs,
  author = {Richard Townsend and Adam Tsakalidis and Yiwei Zhou and Bo Wang and Maria Liakata and Arkaitz Zubiaga and Alexandra I. Cristea and Rob Procter},
  title = {{WarwickDCS: From Phrase-Based to Target-Specific Sentiment Recognition}},
  booktitle = {SemEval @ NAACL},
  year = {2015},
  doi = {10.18653/v1/s15-2110},
}
Figure