Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences

Hong Yu, Vasileios HatzivassiloglouEMNLP 2003

Abstract

This study addresses the challenge of distinguishing objective facts from subjective opinions in automated Question Answering (QA) systems. By implementing a multi-level classification framework—spanning documents, sentences, and sentiment polarity—the researchers demonstrate that "weak" metadata can effectively train robust models. The system achieves high precision in document-level filtering and sentence-level sentiment detection, providing a foundation for handling complex, perspective-driven queries.

Introduction

Early Question Answering (QA) systems primarily targeted "factoid" questions with singular, verifiable answers. However, real-world information needs often involve complex topics, such as the motivations behind geopolitical conflicts, which require synthesizing multiple viewpoints. A critical hurdle in this domain is the nuance of subjectivity: a "news" article may contain subjective quotes, while an "editorial" often relies on factual premises. This paper proposes a tiered approach to identify subjective content and determine its polarity (positive or negative).

Methodology

The researchers developed a three-stage pipeline to process text:

  1. Document Classification: A supervised Naive Bayes classifier was employed to categorize articles into "Opinion" (e.g., Editorials, Letters) or "Fact" (e.g., News, Business). Training was facilitated by existing section metadata from the Wall Street Journal.
  2. Sentence Subjectivity Detection: To identify opinionated sentences within documents, the authors compared three methods: a similarity-based approach (SIMFINDER), a standard Naive Bayes classifier using n-grams and POS tags, and a novel "Multiple Naive Bayes" iterative cleaning process. The latter filters out sentences that do not align with their document-level labels to refine the training set.
  3. Polarity Classification: The system determines whether an opinion is positive or negative by calculating log-likelihood ratios for words based on their co-occurrence with seed adjectives. Crucially, the authors expanded the feature set to include nouns, verbs, and adverbs.

Experimental Results

Document-Level Performance

The document classifier demonstrated exceptional performance in distinguishing between factual news and subjective editorials. As shown in the table below, the system achieved F-measures as high as 0.97 when comparing combined categories.

F-measure for document classification across different category groupings.

Sentence-Level Subjectivity and Polarity

At the sentence level, the iterative "Multiple Naive Bayes" approach reached 86–91% accuracy. For polarity classification, the inclusion of diverse parts of speech significantly boosted performance. The researchers tested various "seed" sets to anchor the sentiment analysis, finding that larger seed sets generally improved the precision-recall trade-off.

Precision-recall curves for polarity classification using different seed word sets.

The results indicate that while identifying subjectivity is challenging, the system achieved 90% accuracy in polarity detection, far exceeding the 48% baseline.

Discussion: The Subjectivity Gap

A key insight from the study is the inherent difficulty of the task, even for human annotators. While humans agreed on the broad "fact vs. opinion" distinction approximately 82% of the time, agreement on specific sentiment orientations (positive, negative, neutral) dropped significantly. This highlights the subtle, context-dependent nature of subjective language.

Furthermore, the authors demonstrated the efficacy of "weak labels." By leveraging document-level metadata to bootstrap sentence-level training, they bypassed the need for exhaustive manual annotation, proving that large-scale, noisy data can be refined into a high-quality training resource.

Conclusion

This research provides a scalable framework for sentiment analysis in QA systems. While the current model is highly effective at identifying the presence and polarity of opinions, future work is needed to identify the source of the opinion and the specific entity being discussed. Such advancements would allow systems to cluster perspectives and provide more nuanced summaries for complex queries.

Summary updated

BibTeX
@inproceedings{yu2003towards,
  author = {Hong Yu and Vasileios Hatzivassiloglou},
  title = {{Towards Answering Opinion Questions: Separating Facts from Opinions and Identifying the Polarity of Opinion Sentences}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2003},
}
Figure