Acquiring Commonsense Knowledge for Sentiment Analysis through Human Computation

Marina Boia, Claudiu Cristian Musat, Boi FaltingsAAAI 2014 · DOI

Abstract

Sentiment analysis often struggles with context-dependency, where the polarity of a word changes based on its surroundings. This paper presents Guesstiment, a crowdsourced game that leverages human computation to capture "commonsense" contextual knowledge. By transforming the identification of word-context pairs into an engaging task, the authors developed a context-dependent lexicon. Experimental results demonstrate that this lexicon significantly outperforms traditional sentiment dictionaries and standard machine learning models in both source and target domains.

Introduction

The primary challenge in sentiment analysis is the "commonsense gap." While words like "excellent" are universally positive, many descriptors are ambiguous without context. For instance, "small" is a desirable trait for a camera but a negative one for a hotel room. Traditional lexicon-based approaches use static dictionaries that ignore these nuances, while large-scale machine learning models often fail to generalize across different domains without prohibitive amounts of training data. Guesstiment addresses this by eliciting intuitive human knowledge through gamification.

Methods: The Guesstiment Game

Guesstiment was deployed on Amazon Mechanical Turk to collect high-quality annotations. The game loop requires workers to select a sentiment-carrying phrase, identify an optional disambiguating context word, and label the resulting polarity.

The Guesstiment game interface, featuring a puzzle-based reward system and annotation fields.

To maintain engagement and data integrity, the system employs several strategies:

  • Gamification: Workers solve "riddles" by submitting answers, which gradually reveals a hidden image.
  • Bayesian Scoring: Points are awarded based on agreement with the model's current belief and the novelty of the information provided.
  • Quality Control: "Gold standard" rounds with known answers are used to filter out low-quality inputs.

Results and Evaluation

The researchers evaluated the resulting context-dependent lexicon (CLgame) against established baselines, including the Bing Liu lexicon and Support Vector Machine (SVM) classifiers.

Comparison of sentiment analysis accuracy across various domains: electronics, services, books, and movies.

Key findings include:

  • Source Domain Superiority: On vacuum and camera reviews, CLgame achieved 85.17% accuracy, a 16.8% improvement over context-independent lexicons.
  • Cross-Domain Robustness: As shown in the performance comparison, the game-derived lexicon consistently outperformed the SVM and traditional lexicons across diverse domains like books and services. In the "Books" category, it surpassed the SVM by 12.8%.
  • Improved Recall: CLgame demonstrated a 22.5% boost in recall on hotel reviews compared to previous automated context-aware methods, capturing more nuanced expressions.

Discussion

The success of Guesstiment highlights that while context combinations are vast, the number of meaningful exceptions to base polarities is manageable for human annotators. This "commonsense" knowledge is highly portable across industries. Future work may explore hybrid models that use human-elicited seeds to improve automated machine learning techniques and expand the approach to non-English languages.

Summary updated

BibTeX
@inproceedings{boia2014acquiring,
  author = {Marina Boia and Claudiu Cristian Musat and Boi Faltings},
  title = {{Acquiring Commonsense Knowledge for Sentiment Analysis through Human Computation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8840},
}
Figure