TL;DR
This paper presents a supervised learning-to-rank system for predicting sentiment intensity of words and phrases, achieving first place in the SemEval-2016 Task 7 for English domains. The system leverages a rich set of features, including sentiment lexicons, word vectors, and linguistic cues, demonstrating the effectiveness of a ranking approach over traditional regression for this task.
What Problem Does This Paper Address?
Sentiment analysis is a crucial area in Natural Language Processing, with sentiment lexicons playing a vital role. However, many existing lexicons only provide a general positive or negative orientation, or discrete labels like "strong" or "weak." There's a clear need for systems that can automatically assign continuous, real-valued scores indicating the precise intensity of sentiment for words and phrases. This is particularly challenging for informal text like Twitter, which contains slang, misspellings, and complex phrasal structures. SemEval-2016 Task 7 aimed to address this by tasking participants with predicting sentiment intensity scores between 0 and 1 for English and Arabic terms.
Methods
The authors propose a supervised learning-to-rank system, framing sentiment intensity prediction as a ranking problem rather than a regression one. Their system comprises four main modules:
- Query Reconstruction: The training data (word lists with sentiment scores) is transformed into "queries" to facilitate the ranking process.
- Data Preprocessing: This module handles the noisy nature of Twitter data by normalizing informal words (e.g., "waiiiiit" to "wait") and hashtags, followed by lemmatization and stemming.
- Feature Engineering: A comprehensive set of features is extracted:
- Sentiment Lexicon Features: Scores from seven different sentiment lexicons (e.g., Bing Liu, MPQA, AFINN) are incorporated, considering maximum, minimum, and sum of scores for words within a phrase. Negations are handled by reversing sentiment orientation.
- Word Vectors: Both traditional word vectors (GoogleW2V, GloVe) and sentiment-specific word vectors (SWV, SSWE) are used. For phrases, they either use pre-trained vectors if the phrase exists in training data or sum the vectors of individual words.
- Linguistic Features: Features like "Negation" (presence of negation words) and "Emphasize" (repeated characters) are included to capture nuances.
- Ranking Model: A pair-wise learning-to-rank approach is employed. Records are paired, and the model learns to predict if one record is more relevant (higher sentiment intensity) than another. They experimented with various ranking algorithms and settled on Random Forest for their final systems.
Datasets
The primary datasets used were from SemEval-2015 Task E (Sentiment Analysis on Tweet), specifically the 15trial and 15test data, which were combined for training. Additionally, the Language Assessment by Mechanical Turk (LabMT) lexicon was used to expand the training data. LabMT contains over 10,000 words rated on a 1-9 scale, which was min-max normalized to a 0-1 scale to match the task requirements. Development data was provided by the organizers for each domain (16trial_Twitter and 16trial_General).
Key Insights and Ideas
- Ranking is Superior to Regression: The paper strongly advocates for treating sentiment intensity prediction as a ranking problem, showing that their ranking-based system outperforms regression methods. This suggests that the relative ordering of sentiment scores is more important than predicting an exact score.
- Sentiment Lexicons are Crucial: Sentiment lexicon features were found to be highly contributory, indicating that pre-existing knowledge about word sentiment is a strong signal for intensity.
- Sentiment Word Vectors Capture Nuance: Sentiment-specific word vectors (like SWV) performed significantly better than traditional word vectors, highlighting their ability to encode sentiment information directly, which is vital for this task.
- Twitter Data is Distinct: The performance gap between the English Twitter domain and the General English domain was attributed to the nature of Twitter data, which often features phrases with mixed or opposite polarities (e.g., "happy accident").
Key Results
The ECNU system achieved 1st place in both the General English and English Twitter domains of SemEval-2016 Task 7.
- General English Domain: Achieved a Kendall correlation coefficient of 70.42% (Spearman: 86.27%).
- English Twitter Domain: Achieved a Kendall correlation coefficient of 52.31% (Spearman: 67.40%).
These results demonstrate the effectiveness of their supervised learning-to-rank approach and the chosen feature set.
Limitations and Future Work
- Phrase Representation: The authors note that their method for representing phrase vectors (summing word vectors for unseen phrases) was not as effective as expected. They suggest that exploring more sophisticated ways to represent phrases is an interesting direction for future work.
- Data Diversity: The performance difference between the two domains suggests that more robust methods for handling diverse and informal language structures, particularly those with mixed polarities, are needed.
- Feature Selection: While they used a feature selection method (hill climbing), further exploration into optimal feature combinations could yield even better results.
Summary updated