Recognizing Opinion Sources Based on a New Categorization of Opinion Types

Lingjia Deng, Janyce WiebeIJCAI 2016

TL;DR

This research introduces a novel framework for identifying opinion sources by categorizing opinions as either participant or non-participant. By integrating phrase-level embeddings, transductive learning, and Probabilistic Soft Logic (PSL), the model achieves state-of-the-art performance on the MPQA corpus, significantly outperforming traditional models that treat all opinion types uniformly.

What Problem Does This Paper Address?

In fine-grained sentiment analysis, identifying the "source"—the entity holding the sentiment—is crucial. Traditional approaches typically categorize sources as either the writer or a specific noun phrase (NP) within the text. However, this binary classification often fails to capture the nuanced relationship between an opinion expression and its holder.

Consider the difference between "Tom criticized the student" and "He embezzled the pension." In the first case, Tom is both the agent and the source of the criticism. In the second, while "He" is the agent of the action, the writer is the source of the negative opinion regarding the embezzlement. Previous models relying heavily on Semantic Role Labeling (SRL) often misidentify the agent as the source in cases where the agent is actually the target of the writer's criticism.

Examples of opinion expressions, their sources, and the proposed categorization.

Methods

The authors propose a two-stage approach to bridge this gap:

  1. Opinion Categorization: Opinions are classified into two types:
  2. Participant Opinions: The source is a direct participant in the event (e.g., the agent of "criticize").
  3. Non-participant Opinions: The source is external to the event (e.g., the writer or a reporting speaker like "Mary says..."). To handle the lack of labeled data for these categories, the authors use unfolding Recursive Autoencoders (uRAE) for 100-dimensional phrase embeddings and a Transductive SVM (TSVM). This allows the model to leverage unlabeled data to refine its decision boundaries based on silver-standard heuristics.

  4. Joint Source Recognition via PSL: Instead of a rigid pipeline, the system utilizes Probabilistic Soft Logic (PSL). PSL allows for the definition of first-order logic rules with continuous truth values. This framework enables the model to jointly optimize for the most probable source by weighing evidence from CRFs, semantic agents, and syntactic subjects.

Datasets

The model was evaluated on the MPQA 2.0 corpus, a standard benchmark for fine-grained sentiment analysis. The dataset was split into 135 documents for development and 400 documents for 10-fold cross-validation. The study focused on explicit sources anchored to the writer or specific noun phrases, excluding approximately 5% of opinions with implicit sources.

Key Insights and Ideas

  • Contextual Role Dependency: The semantic role of a source is intrinsically linked to the type of opinion expression. Action-oriented expressions (e.g., "criticize") typically point to participants, while descriptive expressions (e.g., "considerate") often point to non-participants.
  • Representation Learning: uRAE embeddings effectively capture semantic nuances (e.g., distinguishing "criticized" from "embezzled") without the need for manual feature engineering like N-grams or Part-of-Speech tags.
  • Probabilistic Logic: PSL provides a robust way to encode linguistic constraints (e.g., mutual exclusivity of opinion types) while managing the inherent uncertainty of natural language processing.

Key Results

The proposed Joint model demonstrated significant improvements over existing methods. On the MPQA dataset, it achieved an F1-measure of 63.75% (exact match) and 64.31% (overlap match).

Performance breakdown of the Joint model and its components.

As shown in the results, the Joint model significantly outperformed the previous state-of-the-art ILP-based model (61.63% F1) and a standard pipeline approach (50.84% F1). Furthermore, the use of Transductive SVMs provided a statistically significant boost over standard non-transductive SVMs (63.75% vs 61.47% F1), proving that leveraging unlabeled data helps bridge the gap when specific labels are missing.

Limitations and Future Work

  • Implicit Sources: The model currently ignores implicit sources where the source isn't explicitly named in the sentence. Future work aims to resolve these to entities mentioned elsewhere in the document.
  • Pipeline Dependencies: The system relies on external tools like SENNA for semantic role labeling. Errors in these upstream tools can propagate to the PSL model.
  • Joint Extraction: While this paper focuses on recognizing sources for given expressions, future work should aim to jointly extract expressions, sources, and targets simultaneously using PSL.

Summary updated

BibTeX
@inproceedings{deng2016recognizing,
  author = {Lingjia Deng and Janyce Wiebe},
  title = {{Recognizing Opinion Sources Based on a New Categorization of Opinion Types}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure