Identifying Opinionated Sentences

Theresa Wilson, David R. Pierce, Janyce WiebeNAACL 2003

TL;DR

This paper presents a system for automatically identifying sentences that express opinions, beliefs, emotions, or evaluations within news articles. This is crucial for applications like question answering and summarization that need to understand subjective content, not just factual reporting. The system achieves high precision in identifying these opinionated sentences, demonstrating its practical utility.

What Problem Does This Paper Address?

In the realm of natural language processing, particularly for tasks like question answering and text summarization, distinguishing between factual statements and subjective expressions is a significant challenge. News articles, while primarily reporting facts, often intersperse these with opinions, emotions, and evaluations, especially when discussing controversial topics. This makes it difficult for automated systems to accurately gauge public sentiment or answer questions that require understanding subjective viewpoints. The authors address this by developing a system that can reliably pinpoint sentences containing such opinionated content.

Methods

The system's architecture involves several key steps. First, it utilizes the GATE (General Architecture for Text Engineering) framework to perform standard NLP preprocessing tasks: tokenization, sentence splitting, and part-of-speech tagging. Following this, the document is stemmed, and then searched for features indicative of opinionated language. Finally, these identified features are used to classify sentences as opinionated or not. The system leverages a combination of manually identified features, drawing from linguistic resources like FrameNet and verb class studies, and automatically identified features. These automatic features are learned using information extraction techniques applied to a large corpus of unannotated world news documents, a common approach for scaling up feature discovery.

Datasets

The system was evaluated on a corpus of 109 documents, comprising approximately 2200 sentences, sourced from world news. These articles were specifically annotated for expressions of opinions as part of a summer workshop on multi-perspective question answering. In this test corpus, a significant 59% of sentences were identified as opinionated.

Key Insights and Ideas

  • The ubiquity of opinion in news: Even in seemingly objective news reports, opinionated language is prevalent, making its automatic detection a valuable capability.
  • The trade-off between recall and precision: The system can be tuned to be highly precise (identifying only sentences it's very sure about) at the cost of missing some opinionated sentences (lower recall), or it can be set for higher recall (capturing more opinionated sentences) but with a higher rate of false positives. This highlights a common challenge in classification tasks.
  • Leveraging both manual and automatic feature engineering: Combining linguistic knowledge with data-driven extraction techniques appears to be a robust strategy for building effective opinion detection systems.

Key Results

The paper reports on the performance of the opinionated sentence identification component. The system can be tuned to achieve very high precision, reaching 91% precision while identifying only those sentences it is very sure about. Alternatively, it can be configured for higher recall, achieving 77% recall by identifying more opinionated sentences, though this comes with a trade-off of increased errors.

Limitations and Future Work

The authors acknowledge that the system can be tuned for either high precision or high recall, implying that achieving both simultaneously remains a challenge. While they don't explicitly detail future work in the provided excerpt, the nature of the problem suggests that expanding the types of opinions detected, improving robustness across different genres of text, and integrating this capability into larger NLP applications are natural next steps. The reliance on specific annotation guidelines from the workshop also implies that adapting the system to different annotation schemes or unsupervised learning scenarios could be a future direction.

Summary updated

BibTeX
@inproceedings{wilson2003identifying,
  author = {Theresa Wilson and David R. Pierce and Janyce Wiebe},
  title = {{Identifying Opinionated Sentences}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}
Figure