Mining Comparative Sentences and Relations

Nitin Jindal, Bing LiuAAAI 2006

Introduction

While traditional sentiment analysis focuses on identifying whether a user likes or dislikes a single entity, real-world decision-making often relies on comparisons. Consumers frequently ask, "Is Product X better than Product Y?" rather than just "Is Product X good?" This paper introduces Comparative Sentence Mining (CSM), a framework designed to identify comparative sentences and extract the specific relations between entities, features, and comparative keywords.

The Challenge of Comparative Sentences

Comparative sentences are linguistically distinct from standard opinionated text. They typically involve multiple entities, specific features (e.g., "battery life"), and an ordering relation (superiority, inferiority, or equality). A major hurdle is that simple keyword matching is insufficient; for instance, the sentence "I cannot agree with you more" contains the comparative keyword "more" but does not actually compare two entities. The authors address two primary tasks:

  1. Identification: Distinguishing true comparative sentences from non-comparative ones and classifying their type.
  2. Extraction: Identifying the entities being compared, the feature involved, and the comparative keyword.

Methodology: Sequential Rule Mining

The authors propose moving beyond standard "bag-of-words" classification toward Sequential Rule Mining to capture the structural regularities of comparisons.

Class Sequential Rules (CSR)

CSRs are used to identify comparative sentences. The system identifies patterns of words and Part-of-Speech (POS) tags surrounding 83 "pivot" keywords (such as more, -er adjectives, and less). These patterns are then used as features for a Naive Bayes classifier to distinguish comparisons from non-comparative uses of the same keywords.

Label Sequential Rules (LSR)

For the more complex task of relation extraction, the authors introduce Label Sequential Rules (LSRs). Unlike traditional Information Extraction (IE) methods that rely on local context, LSRs utilize wildcards and labels (e.g., $entityS1`, `$feature) to capture long-range dependencies within a sentence. This allows the system to recognize that in English, the primary entity often precedes the comparison word, even if several words lie between them.

Dataset and Evaluation

The research utilized a dataset compiled from diverse web sources, including consumer reviews, forum postings, and news articles.

Distribution of sentence types within the manually labeled dataset.

As shown in the table above, the dataset was manually labeled by two human judges to establish a gold standard. It includes a significant number of "Other sentences" to test the system's ability to filter out non-comparative text.

Key Results

The proposed methods demonstrated significant improvements over baseline models.

  • Identification: The CSR + Naive Bayes approach achieved an F-score of 81% for identifying gradable comparative sentences.
  • Type Classification: Once identified, classifying the comparison type (gradable, equative, or superlative) was highly accurate, with SVMs reaching 96% accuracy.
  • Extraction Performance: The most notable result was the performance of LSRs in the extraction task.
Comparison of extraction performance (Precision, Recall, F-Score) between LSRs and CRF models.

As illustrated in Figure 1, LSRs significantly outperformed Conditional Random Fields (CRF), particularly in extracting entities and features. While CRFs are standard for many sequence labeling tasks, they struggled with the varied positioning of entities in comparative sentences. LSRs achieved a total F-score of 72%, compared to 58% for the CRF baseline, and reached 100% precision for identifying the primary entity (Entity S1).

Conclusion and Future Work

This work demonstrates that sequential pattern mining is a powerful tool for understanding the structured nature of comparative language. By capturing global sentence patterns, LSRs provide a robust mechanism for extracting complex relations that traditional models miss.

Future research directions include:

  • Expanding the framework to handle non-gradable comparisons (e.g., "Phone A has a stylus, but Phone B does not").
  • Implementing anaphora resolution to handle pronouns (e.g., "It is better than the Nikon").
  • Adapting the model to handle multiple relations within a single complex sentence.

Summary updated

BibTeX
@inproceedings{jindal2006mining,
  author = {Nitin Jindal and Bing Liu},
  title = {{Mining Comparative Sentences and Relations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure