Identifying Expressions of Opinion in Context

Eric Breck, Yejin Choi, Claire CardieIJCAI 2007 · OpenReview

Abstract

While traditional information extraction systems have been built to answer questions about facts, subjective information extraction systems will answer questions about feelings and opinions. A crucial step towards this goal is identifying the words and phrases that express opinions in text. Indeed, although much previous work has relied on the identification of opinion expressions for a variety of sentiment-based NLP tasks, none has focused directly on this important supporting task. Moreover, none of the proposed methods for identification of opinion expressions has been evaluated at the task that they were designed to perform. We present an approach for identifying opinion expressions that uses conditional random fields and we evaluate the approach at the expression-level using a standard sentiment corpus. Our approach achieves expression-level performance within 5% of the human interannotator agreement.

Introduction

This foundational 2007 research from Cornell University shifts the focus of sentiment analysis from document-level classification to fine-grained expression extraction. While early work often categorized entire sentences as "positive" or "negative," Breck, Choi, and Cardie (2007) aimed to identify the specific spans of text—individual words and phrases—that carry subjective meaning.

The Challenge of Subjective Information Extraction

Traditional Information Extraction (IE) typically focuses on factual relationships (e.g., Who did what to whom?). In contrast, "Subjective IE" seeks to understand the underlying sentiment: How does an entity feel about a topic?

Identifying these "anchors" of subjectivity is complex because sentiment is highly context-dependent. For instance, the word "said" is usually a neutral reporting verb, but in the context of "He said the result was a 'highway robbery'," it becomes part of a subjective expression. Previous methods relied on static "subjectivity clues," which often failed to capture the creative and context-heavy nature of human opinion.

Methodology: Conditional Random Fields

The authors frame opinion identification as a sequence-tagging task using Conditional Random Fields (CRFs), specifically a linear-chain model. CRFs are particularly effective here because they model the dependencies between adjacent labels, considering the broader context of surrounding words.

Encoding and Feature Engineering

A key design choice involved the encoding scheme. While many NLP tasks utilize IOB (Inside/Outside/Beginning) encoding to distinguish between adjacent entities, the authors found that subjective expressions are rarely adjacent. Consequently, they adopted a simpler IO encoding (In/Outside), which proved more effective for this specific task.

Comparison of IOB and IO encoding schemes for opinion expression extraction.

The model incorporates a diverse set of features:

  • Lexical: A +/- 4 word window around the target token.
  • Syntactic: Part-of-speech tags and constituent types (e.g., noun phrase, verb phrase) derived from a partial parser.
  • Dictionary-based: Integration of WordNet hypernyms, Levin verb classes, FrameNet categories, and existing subjectivity lexicons.

Dataset: The MPQA Corpus

The approach was evaluated using the MPQA (Multi-Perspective Question Answering) Corpus, which contains 535 news documents. Annotations distinguish between two types of expressions:

  • Direct Subjective Expressions (DSEs): Explicit attitudes, such as "criticized" or "faithful to."
  • Expressive Subjective Elements (ESEs): Words or phrases that convey subjectivity through tone, such as "illegitimate" or "highway robbery."

Analysis and Insights

The research highlights several critical aspects of fine-grained sentiment analysis:

  1. The Boundary Problem: Human annotators frequently agree on the presence of a subjective expression but disagree on its exact boundaries (e.g., "criticized" vs. "roundly criticized"). To account for this, the authors utilized "soft" evaluation metrics (overlap) alongside "exact" matching.
  2. Model Complexity: Interestingly, a simpler "Order-0" CRF (which ignores the previous tag's label) often outperformed the "Order-1" model on overlap metrics. This suggests that more complex models might overfit to the inconsistent boundaries in human annotations.
  3. The Impact of WordNet: Ablation studies revealed that WordNet features provided the most significant performance boost. This indicates that general semantic knowledge is more valuable for identifying novel opinion expressions than specialized, narrow subjectivity dictionaries.
Feature ablation study showing the impact of various lexical and semantic resources on model performance.

Key Results

The CRF-based system achieved performance remarkably close to human levels:

  • DSE Identification: Reached an F-measure of 70.65% (overlap metric).
  • ESE Identification: Reached an F-measure of 63.43% (overlap metric).
  • Human Benchmarking: These results are within 5% of the human interannotator agreement (approximately 72-75%), suggesting the model is operating near the theoretical ceiling of the dataset.

Limitations and Future Directions

Despite its success, the model faces challenges with ambiguous speech events (e.g., "said," "statement"). The authors suggest that incorporating features to identify the "source" or "holder" of an opinion could improve disambiguation. Additionally, since subjective expressions often cluster together, future work could explore features that measure local subjectivity "density." Finally, the inherent noise in human boundary annotations remains a hurdle for achieving high precision in exact-match scenarios.

Summary updated

Abstract

While traditional information extraction systems have been built to answer questions about facts, subjective information extraction systems will answer questions about feelings and opinions. A crucial step towards this goal is identifying the words and phrases that express opinions in text. Indeed, although much previous work has relied on the identification of opinion expressions for a variety of sentiment-based NLP tasks, none has focused directly on this important supporting task. Moreover, none of the proposed methods for identification of opinion expressions has been evaluated at the task that they were designed to perform. We present an approach for identifying opinion expressions that uses conditional random fields and we evaluate the approach at the expression-level using a standard sentiment corpus. Our approach achieves expression-level performance within 5% of the human interannotator agreement.
BibTeX
@inproceedings{breck2007identifying,
  author = {Eric Breck and Yejin Choi and Claire Cardie},
  title = {{Identifying Expressions of Opinion in Context}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2007},
  url = {https://openreview.net/forum?id=Sy4AHNM_br},
}
Figure