AKTSKI at SemEval-2016 Task 5: Aspect Based Sentiment Analysis for Consumer Reviews

Shubham Pateria, Prafulla ChoubeySemEval @ NAACL 2016 · DOI

TL;DR

This paper presents AKTSKI's approach to aspect-based sentiment analysis (ABSA) for SemEval-2016 Task 5. They introduce a novel method that leverages the sequential nature of sentiment within reviews, showing a slight but promising improvement in accuracy by considering how preceding sentences influence the sentiment of the current one.

What Problem Does This Paper Address?

In today's digital age, businesses heavily rely on online reviews to understand customer opinions. However, reviews often contain mixed sentiments about different aspects of a product or service (e.g., "great food, but terrible service"). Aspect-Based Sentiment Analysis (ABSA) aims to pinpoint the sentiment expressed towards specific aspects within a review. This is more nuanced than overall sentiment analysis because it requires identifying both the aspect and its associated sentiment. The SemEval-2016 Task 5 specifically focused on this challenge, building upon previous ABSA tasks. The authors tackle the problem of accurately predicting sentiment polarity for identified aspects, especially in cases where context from previous sentences is crucial for disambiguation.

Methods

The AKTSKI system is built upon a Support Vector Machine (SVM) with a Radial Basis Function (RBF) kernel. They employ a multi-stage approach:

  • Pre-processing: Reviews are parsed into sentences, and each sentence is further broken down based on identified aspects. This involves using a Stanford Dependency Parser to identify terms closely related to the aspect target.
  • Feature Engineering: A comprehensive set of features is extracted, including:
  • Sentiment lexicons (Bing Liu, MPQA, SentiWordNet).
  • Handling of negation terms to modify lexicon scores.
  • Uni-grams and bi-grams from sentence segments.
  • A self-generated list of "neutral" terms.
  • Punctuation (specifically '!') as a potential indicator of positive sentiment.
  • Keywords associated with specific aspect categories (e.g., "high" and "low" for "FOOD#PRICES").
  • Sequence Pattern Analysis (Experimental): This is the core novelty. They observe that reviews often exhibit a "Flow" (consistent sentiment) or "Trans" (sentiment transition) pattern. Two methods are proposed to exploit this:
  • Method 1: Creates a new feature set based on contrast keywords and sentiment keywords present in a sentence, considering the influence of previous sentences.
  • Method 2: A more auto-regressive approach that uses initial predictions from the basic model and then iteratively refines them by considering the influence of preceding predicted polarities. This method is preferred for its theoretical flexibility.

Datasets

The system was trained and evaluated on the dataset provided for SemEval-2016 Task 5, specifically for the Restaurant domain in English. This dataset already contains pre-identified aspects within reviews, allowing the system to focus solely on polarity detection.

Key Insights and Ideas

  • Sequential Context Matters: The most significant insight is that sentiment in one sentence can be heavily influenced by sentiments expressed in preceding sentences within the same review. This "Flow or Trans" pattern is crucial for disambiguating ambiguous sentiments.
  • Disambiguation through Context: For example, a sentence that might be classified as neutral by a basic model could be correctly identified as negative if a preceding sentence clearly indicates a negative overall experience.
  • Neutral Terms are Important: The authors found that explicitly defining a list of "neutral" terms, which don't neatly fit into positive or negative lexicon categories, can improve performance.
  • Punctuation as a Signal: The observation that exclamation marks often co-occur with positive sentiment, while a minor feature, highlights the value of exploring seemingly trivial linguistic cues.

Key Results

  • The AKTSKI system achieved an accuracy of 0.717 on the evaluation dataset using Method 2, which incorporates polarity sequence patterns.
  • When evaluated against gold-labeled data after the competition, the accuracy without sequence patterns was 0.668. Using Method 1 improved this to 0.702, and Method 2 further increased it to 0.717.
  • These improvements, while described as "small increments," demonstrate the potential of sequence-aware approaches in ABSA.

Limitations and Future Work

  • Limited Effort on Feature Engineering: The authors acknowledge that their system might have been outperformed by others due to less effort invested in developing more robust features or ensemble methods.
  • Potential Bias in Keyword Dictionary: The keyword dictionary used for feature generation was created on the full training dataset, which might introduce bias.
  • Tuning for Training Data: Method 2 was tuned on the training data, suggesting it might perform less optimally on unseen datasets.
  • Need for Deeper Investigation: The authors emphasize that the polarity sequence pattern approach requires more research to devise more robust feature extraction methods and to handle sequence patterns more effectively.
  • Broader Datasets: Testing the approach on broader datasets is suggested.
  • Ensemble Methods: Exploring ensemble models and multi-class Platt Scaling is also proposed for future work.

Summary updated

BibTeX
@inproceedings{pateria2016aktski,
  author = {Shubham Pateria and Prafulla Choubey},
  title = {{AKTSKI at SemEval-2016 Task 5: Aspect Based Sentiment Analysis for Consumer Reviews}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1051},
}
Figure