Abstract
Sentiment analysis often struggles with context-dependency, where the polarity of a word changes based on its surroundings. This paper presents Guesstiment, a crowdsourced game that leverages human computation to capture "commonsense" contextual knowledge. By transforming the identification of word-context pairs into an engaging task, the authors developed a context-dependent lexicon. Experimental results demonstrate that this lexicon significantly outperforms traditional sentiment dictionaries and standard machine learning models in both source and target domains.
Introduction
The primary challenge in sentiment analysis is the "commonsense gap." While words like "excellent" are universally positive, many descriptors are ambiguous without context. For instance, "small" is a desirable trait for a camera but a negative one for a hotel room. Traditional lexicon-based approaches use static dictionaries that ignore these nuances, while large-scale machine learning models often fail to generalize across different domains without prohibitive amounts of training data. Guesstiment addresses this by eliciting intuitive human knowledge through gamification.
Methods: The Guesstiment Game
Guesstiment was deployed on Amazon Mechanical Turk to collect high-quality annotations. The game loop requires workers to select a sentiment-carrying phrase, identify an optional disambiguating context word, and label the resulting polarity.
To maintain engagement and data integrity, the system employs several strategies:
- Gamification: Workers solve "riddles" by submitting answers, which gradually reveals a hidden image.
- Bayesian Scoring: Points are awarded based on agreement with the model's current belief and the novelty of the information provided.
- Quality Control: "Gold standard" rounds with known answers are used to filter out low-quality inputs.
Results and Evaluation
The researchers evaluated the resulting context-dependent lexicon (CLgame) against established baselines, including the Bing Liu lexicon and Support Vector Machine (SVM) classifiers.
Key findings include:
- Source Domain Superiority: On vacuum and camera reviews, CLgame achieved 85.17% accuracy, a 16.8% improvement over context-independent lexicons.
- Cross-Domain Robustness: As shown in the performance comparison, the game-derived lexicon consistently outperformed the SVM and traditional lexicons across diverse domains like books and services. In the "Books" category, it surpassed the SVM by 12.8%.
- Improved Recall: CLgame demonstrated a 22.5% boost in recall on hotel reviews compared to previous automated context-aware methods, capturing more nuanced expressions.
Discussion
The success of Guesstiment highlights that while context combinations are vast, the number of meaningful exceptions to base polarities is manageable for human annotators. This "commonsense" knowledge is highly portable across industries. Future work may explore hybrid models that use human-elicited seeds to improve automated machine learning techniques and expand the approach to non-English languages.
Summary updated