Chinese Evaluative Information Analysis

Yiou Wang, Jun’ichi Kazama, Takuya Kawada, Kentaro TorisawaCOLING 2012

TL;DR

This paper introduces the Chinese Evaluative Information Analyzer (CEIA), a system designed to extract a richer set of evaluative information from Chinese web data than previous methods. By developing a new annotated corpus and sentiment dictionary, and introducing novel dependency, semantic class, and distance features, CEIA significantly improves the accuracy of identifying evaluative expressions and their targets.

What Problem Does This Paper Address?

The explosion of Chinese web content has led to a massive increase in user-generated opinions, making the analysis of this data a critical task. However, existing research on Chinese opinion and sentiment analysis lags behind that for languages like English, lacking robust tools to handle the diverse topics and writing styles found online. This paper aims to bridge this gap by developing a comprehensive system, CEIA, capable of identifying and analyzing a wider range of evaluative information, including implicit opinions and objective evaluations, which are often overlooked by traditional methods.

Methods

CEIA employs a multi-stage process to analyze Chinese evaluative information:

  • Evaluative Expression Extraction: This core task uses a sequence tagging method with BIO (Beginning, Inside, Outside) tags, trained using Conditional Random Fields (CRFs). Novel features introduced include:
  • Dependency Features: Leveraging syntactic dependency information from a Chinese dependency parser, including head-dependent relations (near, medium, far) and dependency relation labels.
  • Semantic Class Features: Utilizing Brown clustering to represent words by their semantic class, using both full strings and 6-bit prefixes.
  • Distance Features: Encoding the word count distance between a word and the evaluative expression.
  • Evaluation Target Extraction: Similar to expression extraction, this uses BIO tagging with CRFs, incorporating dependency relation features and distance features to identify the entities being evaluated.
  • Evaluation Holder Identification: This involves a two-step process: first, a Support Vector Machine (SVM) to determine if the holder is the author or an external entity, and second, a CRF for extraction if the holder is not the author.
  • Evaluation Type Determination: This task uses one-versus-rest multi-class SVMs, incorporating features like word and POS tag unigrams/bigrams, and a type category dictionary.
  • Sentiment Polarity Determination: A dependency tree-based method using CRFs with hidden variables is employed, similar to prior work on Japanese and English, to classify sentiment as positive or negative.

Datasets

  • Chinese Evaluative Information Corpus: A newly constructed corpus of Chinese sentences extracted from web pages across 66 diverse topics. This corpus was annotated with evaluative expressions, holders, targets, types, and sentiment polarities. It contains 6,680 sentences and 5,111 evaluative information sets.
  • Chinese Sentiment Dictionary: Built semi-automatically by translating a Japanese sentiment dictionary and augmenting it with words from the Chinese Gigaword corpus. It contains 28,411 entries.
  • Chinese Gigaword Version 2.0 (XIN_CMN portion): Used for generating semantic classes, with approximately 311 million words.

Key Insights and Ideas

  • Beyond Explicit Opinions: The paper argues for the importance of analyzing "implicit" evaluations (e.g., requests, objective facts that imply sentiment) alongside explicit subjective expressions, broadening the scope of opinion mining.
  • Syntactic and Semantic Features Matter: The introduction of dependency and semantic class features significantly improves performance, demonstrating that deeper linguistic analysis is crucial for accurate evaluative information extraction.
  • Distance as a Feature: The simple yet effective distance feature between words and evaluative expressions proves highly beneficial for target extraction, highlighting the proximity heuristic in opinion analysis.
  • Fine-grained Analysis: The system's ability to identify evaluative information at a fine-grained level (5-tuple: expression, holder, target, type, polarity) provides a more nuanced understanding of opinions.

Key Results

  • Evaluative Expression Extraction: The system achieved an F-measure of 0.4832 using exact match and 0.5734 using partial match. The partial match performance is considered good, with a precision of 0.7728, making it useful for real-world applications.
  • Evaluation Target Extraction: Achieved an F-measure of 0.6300 with span partial match.
  • Evaluation Holder Identification: Achieved an F-measure of 0.8474 with partial match.
  • Evaluation Type Determination: Accuracy of 0.5787.
  • Sentiment Polarity Determination: Accuracy of 0.8146.
  • New Features Impact: Dependency features showed improvement in recall and precision for expression extraction. Semantic class features (especially full string) were effective, and a combination of features yielded the best results. For target extraction, distance features were most effective, and dependency relation features also showed positive impact.

Limitations and Future Work

  • Exact Match Difficulty: The paper acknowledges that achieving high performance with exact match for evaluative expression spans is challenging due to the difficulty in precisely defining their boundaries.
  • Comparative Sentences: The system currently does not deal with comparative sentences that do not show clear sentiment polarities.
  • Holder Identification: While the system handles cases where the holder is the author or explicitly stated, it notes that in more than half of Chinese cases, holders are not clearly stated, suggesting room for improvement.
  • Future Work: The authors suggest that the system's performance can be further improved by exploring more sophisticated dependency features and semantic class representations. The robustness of the system across diverse domains and writing styles is a key strength, implying potential for broader application.

Summary updated

BibTeX
@inproceedings{wang2012chinese,
  author = {Yiou Wang and Jun’ichi Kazama and Takuya Kawada and Kentaro Torisawa},
  title = {{Chinese Evaluative Information Analysis}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
}
Figure