Modifying SO-PMI for Japanese Weblog Opinion Mining by Using a Balancing Factor and Detecting Neutral Expressions

Guangwei Wang, Kenji ArakiNAACL 2007

TL;DR

This paper adapts the SO-PMI algorithm for Japanese weblog opinion mining, addressing its tendency to produce biased results. By introducing a balancing factor and a method for detecting neutral expressions, the authors significantly improve the algorithm's accuracy and balance for Japanese text.

What Problem Does This Paper Address?

The internet is awash with user-generated opinions in the form of weblogs, product reviews, and forum posts. Extracting and analyzing these opinions automatically is crucial for businesses and researchers alike. However, processing this unstructured, free-form text is challenging. The SO-PMI (Semantic Orientation Using Pointwise Mutual Information) algorithm, a successful unsupervised method for English opinion mining, struggles when directly applied to Japanese. Initial attempts resulted in heavily skewed sentiment classifications, with positive opinions being overwhelmingly favored, rendering the algorithm practically useless for Japanese data. This paper aims to bridge this gap by modifying SO-PMI to effectively handle the nuances of Japanese opinion expressions.

Methods

The core of this work involves adapting and enhancing the SO-PMI algorithm.

  • Initial Adaptation: The authors first translated the SO-PMI formula to Japanese, using the search engine operator "AND" instead of the "NEAR" operator used in English versions (as "NEAR" is not well-supported for Japanese). This initial step, however, led to extremely biased results.
  • Reference Set Expansion: To address the bias, the single reference words "excellent" and "poor" were replaced with sets of common positive ("p_basic") and negative ("n_basic") Japanese words. This aimed to provide a more robust representation of positive and negative sentiment.
  • Balancing Factor: Recognizing that the sheer volume of search engine hits for positive vs. negative reference words could still cause imbalance, a "balancing factor" was introduced. This factor, controlled by a weight 'α', adjusts the SO score based on the ratio of hits for the positive and negative reference sets.
  • Neutral Expression Detection: The authors observed that many neutral phrases were being misclassified as either positive or negative. To combat this, they developed a set of rules to identify and flag neutral expressions. These rules consider the association of a phrase with both positive and negative reference sets, as well as the overall SO score.

Datasets

  • Benchmark Dataset: The authors collected a benchmark dataset consisting of 200 positive and 200 negative Japanese opinion sentences. These sentences were sourced from product reviews on "Kakaku.com," a Japanese website specializing in product comparisons and user reviews. The reviews were hand-labeled as either positive or negative.
  • Related Work Comparison: The paper also references the performance of Turney's original SO-PMI on English data, which achieved an average accuracy of 74% on 410 reviews from Epinions.

Key Insights and Ideas

  • Direct Translation is Insufficient: The most striking insight is that simply translating an algorithm and its components (like reference words and search operators) from one language to another, especially with different linguistic structures and search engine behaviors, is unlikely to yield good results.
  • Reference Sets Matter: Expanding single reference words to sets of words significantly improved the algorithm's ability to capture a broader range of sentiment expressions.
  • The Importance of Balance: The sheer difference in the prevalence of positive vs. negative terms on the web can heavily skew sentiment analysis. Introducing a balancing factor is crucial for achieving a more equitable classification.
  • Neutrality is a Distinct Category: Failing to account for neutral expressions leads to misclassification and reduces overall accuracy. Explicitly detecting and handling neutral statements is vital for a comprehensive sentiment analysis system.

Key Results

  • Initial Japanese SO-PMI (Test 1): Achieved a highly skewed result with 95% positive accuracy (PA) and only 8% negative accuracy (NA).
  • Modification 1 (Reference Sets - Test 2): Using "p_basic" and "n_basic" sets improved NA to 12% but still resulted in a very unbalanced outcome (99% PA).
  • Modification 2 (Balancing Factor - Test 3): With the balancing factor (α=0.9), the results became more balanced, achieving 75% PA and 70% NA.
  • Modification 3 (Neutral Detection - Test 4): The final proposed approach, incorporating neutral expression detection, achieved a well-balanced result of 78% PA and 72% NA. This represents a significant improvement over the initial direct translation and a considerable enhancement in balance compared to earlier modifications.

Limitations and Future Work

  • Limitations:
    • The "NEAR" operator, which was effective for English, does not work well for Japanese, necessitating the use of the less precise "AND" operator.
  • The initial direct translation of SO-PMI to Japanese resulted in extremely poor negative accuracy.
  • Even with modifications, the authors acknowledge that the balance between positive and negative sides was still influenced by the difference in page hits between reference words/sets.
  • Future Work:
    • The authors plan to evaluate different choices of words for the positive and negative reference sets.
  • They also intend to appraise their proposed approach on other languages.

Summary updated

BibTeX
@inproceedings{wang2007modifying,
  author = {Guangwei Wang and Kenji Araki},
  title = {{Modifying SO-PMI for Japanese Weblog Opinion Mining by Using a Balancing Factor and Detecting Neutral Expressions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure