Do We Really Need Lexical Information? Towards a Top-down Approach to Sentiment Analysis of Product Reviews

Yulia Otmakhova, Hyopil ShinNAACL 2015

TL;DR

This paper proposes a "top-down" approach to sentiment analysis of product reviews, arguing that relying solely on lexical sentiment cues is insufficient. By leveraging the overall review score (primary polarity) and discourse patterns, they achieve significantly better results than traditional "bottom-up" lexical methods, suggesting that understanding the narrative structure is key.

What Problem Does This Paper Address?

Traditional sentiment analysis methods for product reviews often rely on "bottom-up" approaches. This means they start by identifying sentiment-bearing words (lexical information) and then try to build up to an overall sentence or document sentiment. However, this approach faces several challenges:

  • Implicit Sentiment: Reviews can express sentiment without using explicit sentiment words (e.g., "online learning at Capella was the most difficult form of education I have undergone!").
  • Context-Dependent Polarity: The sentiment of a word can change based on context or domain.
  • Complex Structures: Sentences with complex grammatical structures or irony can be misclassified.
  • Objective Sentences: Distinguishing between objective statements and subjective opinions is difficult with purely lexical methods.

The authors argue that product reviews are not just a "bag of words" but narratives with underlying structures and patterns. They question whether a purely lexical approach is truly necessary, especially when external information like a review's overall score is available.

Methods

The paper introduces and evaluates a "top-down" approach to sentiment analysis. Instead of starting with individual words, this method begins by inferring the overall sentiment (primary polarity) of a review, typically from its star rating. This primary polarity is then assigned to all segments of the review. Subsequently, the system "chisels out" incorrect sentiment assignments by identifying discourse patterns that signal a change in polarity or indicate objective segments.

The key components of their top-down approach are:

  1. Primary Polarity Classifier: Uses the review's star rating to assign an initial polarity (positive for 4-5 stars, negative for 1-3 stars) to all segments.
  2. Contrast Classifier: Identifies discourse patterns (explicit markers like "but," "although," and implicit hedges) that indicate a shift in sentiment within the review, allowing for correction of misclassified polarities.
  3. Background Classifier: Detects patterns (acquirement, personal background, personal experience) that signal objective sentences, which are then re-labeled as "objective" rather than polar.

For comparison, they also implement and evaluate a "bottom-up" approach. This involves a baseline lexical classifier (Stanford Sentiment Analysis Classifier) and then incrementally adds features: local discourse context (previous/next sentence polarity), contrast features, background features, and finally, primary polarity features.

Datasets

The experiments were conducted using the Filatova's Amazon product reviews corpus (also known as the Sarcasm Corpus). This corpus contains 817 ironic and regular reviews.

  • Training Data: 100 reviews were randomly selected.
  • Test Data: 20 reviews were randomly selected.

To prevent bias from the skewed distribution of review scores (e.g., many 5-star reviews), they used equal-sized random samples across all possible scores for both training and testing. The corpus was also used as a development set for feature extraction.

Key Insights and Ideas

  • Narrative Structure Over Lexical Cues: Product reviews have a predictable narrative structure that can be exploited. The overall sentiment of a review is a strong indicator of the sentiment of most of its sentences.
  • Top-Down is More Effective: Starting with the global sentiment and then refining it with discourse cues ("chiseling out" incorrect bits) is more effective than building up from individual words.
  • Lexical Noise: Purely lexical approaches introduce "noise" and struggle with implicit sentiment, context, and objective statements, leading to unreliable foundations for further analysis.
  • Objective Sentences are Not Neutral: Sentences commonly labeled as "neutral" in other systems are better classified as "objective" in this context, as they don't carry sentiment related to the product.
  • Discourse Cues are Powerful: Contrast and background discourse patterns are crucial for accurately identifying sentiment shifts and objective segments.

Key Results

The top-down approach consistently outperformed the bottom-up approach across various metrics:

  • Bottom-up Approach:
    • Lexical baseline accuracy: 0.6138
    • With all features (including primary polarity): accuracy 0.7241
  • Top-down Approach:
    • Primary polarity classifier accuracy: 0.7379
    • With contrast and background features: accuracy 0.7931

The top-down approach, even with just the primary polarity classifier, achieved better recall and accuracy than any of the bottom-up lexical classifiers. Adding discourse features to the top-down approach further improved performance, leading to a high-precision, high-recall sentiment classifier.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the top-down approach relies on the availability of an external measure of primary polarity (like review scores).
  • The contrast classifier, while improving results, can lead to overestimation of positive segments and lower overall accuracy in some configurations.
  • The background classifier, while improving objective sentence detection, still relies on lexical and grammatical cues, which can be complex.
  • Future Work:
    • The authors believe the top-down classifier can be made even more accurate by incorporating other discourse relations and carefully selected linguistic features.
  • Exploring more complex CRF classifiers and models could further enhance performance.
  • The approach is shown to be effective for texts with predictable structures like product reviews; its applicability to other text types could be investigated.

Summary updated

BibTeX
@inproceedings{otmakhova2015do,
  author = {Yulia Otmakhova and Hyopil Shin},
  title = {{Do We Really Need Lexical Information? Towards a Top-down Approach to Sentiment Analysis of Product Reviews}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure