Building Lexicon for Sentiment Analysis from Massive Collection of HTML Documents

Nobuhiro Kaji, Masaru KitsuregawaEMNLP 2007

TL;DR

This paper tackles the challenge of automatically building sentiment lexicons (lists of positive and negative words/phrases) by leveraging structural clues within Japanese HTML documents. They achieve high precision in identifying polar sentences, compensating for lower recall by using a massive dataset, and then extract polar phrases to build their lexicon.

What Problem Does This Paper Address?

Sentiment analysis aims to understand the emotional tone of text, a fundamental task that often relies on having a comprehensive lexicon of words and phrases that express positive or negative sentiment. Manually creating such lexicons is time-consuming and doesn't scale well. Existing automatic methods, particularly those relying on co-occurrence within a corpus, often struggle with precision. This paper addresses the need for a more accurate and scalable method to automatically build sentiment lexicons, specifically focusing on Japanese text found in HTML documents.

Methods

The authors propose a three-step method:

  1. Extracting Polar Sentences: They exploit "structural clues" within HTML documents, focusing on language and layout structures. This includes:
  2. Language Structure: Identifying patterns like topic markers ("-ha") followed by evaluative words (e.g., "advantage") to pinpoint sentences likely expressing sentiment.
  3. Layout Structure: Utilizing itemizations (lists with headers like "pros" and "cons") and tables (where specific cells indicate positive or negative sentiment) to extract polar sentences. The key here is designing these clues for extremely high precision, even at the cost of recall.

  4. Extracting Polar Phrase Candidates: From the collected polar sentences, they extract candidate polar phrases (primarily adjectives and adjective phrases) and count their occurrences in positive and negative sentences. They focus on main clauses to mitigate issues where negative phrases might appear in overall positive sentences.

  5. Selecting Polar Phrases: They determine a "polarity value" for each candidate phrase using two methods: chi-square and Pointwise Mutual Information (PMI). Based on this value and a threshold, they select phrases to add to their lexicon.

Datasets

  • Training/Extraction Data: A massive collection of one billion Japanese HTML documents. From this, they extracted approximately 500,000 polar sentences (220,716 positive, the rest negative).
  • Evaluation Data: A test set of 405 adjective phrases, manually annotated with polarity (positive, negative, neutral). This set had no overlap with their extracted polar sentence corpus.

Key Insights and Ideas

  • Structural Clues for High Precision: The core innovation is using HTML structure and specific linguistic patterns as high-precision indicators of sentiment, rather than relying solely on co-occurrence. This significantly reduces noise.
  • Massive Data as a Recall Compensator: By accepting lower recall from their precise structural clues, they compensate by processing an enormous volume of HTML documents, ensuring they still gather a substantial corpus of polar sentences.
  • PMI Outperforms Chi-Square: The experimental results show that the PMI-based method for calculating polarity value yields better results than the chi-square method, particularly in terms of recall.
  • Importance of Phrase-Level Analysis: The paper demonstrates the value of extracting phrases, not just single words, for sentiment analysis, as many colloquialisms and multi-word expressions carry sentiment.

Key Results

  • The proposed method achieved high precision in extracting polar sentences, with human judges rating it around 92%.
  • The lexicon construction process, using PMI with a threshold of θ=1.0, achieved the best F-measure.
  • For positive phrases, the best recall-precision curve showed a precision of around 90% at a recall of 80%.
  • For negative phrases, the PMI method achieved similar precision to chi-square but with significantly higher recall.
  • The lexicon successfully learned phrases and colloquial words that are difficult for thesaurus-based approaches.

Limitations and Future Work

  • Neutral Phrases: The authors acknowledge that errors are often related to neutral phrases, and suggest that exploiting a neutral corpus would improve performance. This is identified as a key area for future work.
  • Low Frequency Words: The evaluation indicated that handling low-frequency words remains challenging.
  • Language Specificity: While the core idea of using structural clues is applicable to other languages, the specific patterns identified are Japanese-centric. Future work could explore adapting these ideas to English and other languages.
  • Data Size Impact: The experiments showed that increasing the corpus size significantly improves recall, suggesting that even larger datasets could further enhance performance.

Summary updated

BibTeX
@inproceedings{kaji2007building,
  author = {Nobuhiro Kaji and Masaru Kitsuregawa},
  title = {{Building Lexicon for Sentiment Analysis from Massive Collection of HTML Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure