Exploiting Machine Learning for Comparative Sentences Extraction

Wei Wang, Tie Jun Zhao, Guo Xin, Yong XuInternational Journal of Hybrid Information Technology 2015 · DOI

Abstract: This paper studies the problem of extracting Chinese comparative sentences from user reviews, which is a problem of text classification at the sentence level. The research first addresses the class skewness inherent in review data and then builds a Support Vector Machine (SVM) model to classify comparative and non-comparative sentences on a balanced dataset. Various linguistic and statistical features are introduced to characterize sentences. Experiments conducted on user-generated product reviews demonstrate significant performance, achieving an overall F-score of 85.87%.

Introduction

In the field of sentiment analysis, most research focuses on "direct opinions" (e.g., "This camera is great"). However, "comparative opinions" (e.g., "Camera X is better than Camera Y") often provide more valuable competitive intelligence for both consumers and manufacturers. Identifying these sentences in Chinese text is challenging due to class imbalance—where comparative sentences are rare—and linguistic complexity, as comparisons are not always signaled by obvious keywords.

Methodology and Data Balancing

The researchers proposed a multi-stage approach to tackle these issues. A critical component is the Four-Type Taxonomy, which categorizes sentences based on the presence of comparative indicators and their actual semantic content.

Distribution of sentence types in the dataset, highlighting the prevalence of Type 4 sentences which lack comparative indicators.

To handle class imbalance, the authors used a keyword-based algorithm to filter out "Type 4" sentences (those lacking indicators and clearly non-comparative). This creates a more balanced candidate set for the machine learning model. The remaining sentences were then classified using an SVM with a Radial Basis Function (RBF) kernel.

Feature Engineering

The model relies on three distinct feature types:

  1. Keywords: A lexicon of 102 words and 30 sequences known to signal comparisons.
  2. Class Sequence Rules (CSR): Automated mining of frequent patterns of words and Part-of-Speech (POS) tags using an improved PrefixSpan algorithm.
  3. Manual Rules: 45 hand-crafted patterns designed to capture superlative structures that automated mining often misses.

Experimental Results

The system was evaluated using the COAE 2012 corpus, covering electronic products and automobiles. The dataset contained 1,624 comparative sentences and 7,976 non-comparative sentences.

Performance comparison of different feature combinations, showing the incremental benefit of adding Sequence Patterns (SP) and Manual Rules (MR) to Keywords (KWs).

As shown in the performance analysis, the hybrid model (SVM+KWs+SP+MR) consistently outperformed simpler configurations. While keywords provide a baseline, the inclusion of sequence patterns and manual rules significantly improved the model's ability to identify structural comparisons, leading to a peak precision of 92.21% and an overall F-score of 85.87%.

Discussion and Conclusion

The study reveals that structural sequence patterns are more effective than simple keyword matching for Chinese comparative extraction. While the system achieves high precision, a recall gap remains (approximately 80%), suggesting that some implicit comparisons still evade detection. Future work may involve deeper semantic analysis or modern transformer-based architectures to further refine the extraction of these complex linguistic structures.

Summary updated

Abstract

This paper studies the problem of extracting Chinese comparative sentences from user reviews, which is a problem of text classification in the level of sentence. This paper first deals with the class skewed problem of review data, and then builds a SVM (support vector machine) model to classify comparative and non-comparative sentences into different groups on a balanced dataset. Various linguistic and statistical features are introduced to characterize a sentence. Experiments were conducted on user-generated product reviews. As a result, our experiments show significant performance, an overall Fscore of 85.87%.
BibTeX
@article{wang2015exploiting,
  author = {Wei Wang and Tie Jun Zhao and Guo Xin and Yong Xu},
  title = {{Exploiting Machine Learning for Comparative Sentences Extraction}},
  journal = {International Journal of Hybrid Information Technology},
  year = {2015},
  doi = {10.14257/ijhit.2015.8.3.31},
}
Figure