Sentiment Retrieval using Generative Models

Koji Eguchi, Victor LavrenkoEMNLP 2006

Abstract

Sentiment retrieval represents a significant shift from traditional sentiment classification, focusing on ranking sentences based on both a specific topic and a desired sentiment polarity. This paper introduces a generative language modeling framework designed to address the inherent topic-dependency of sentiment—where the meaning of words like "unpredictable" varies wildly across contexts. By explicitly modeling the relationship between topic-bearing and sentiment-bearing words, the proposed Sentiment Language Model (SLM) achieves up to a 76% improvement over standard retrieval baselines.

Introduction

While sentiment classification (categorizing text as positive or negative) is well-established, sentiment retrieval—finding specific opinions about specific attributes—remains an underserved challenge. A user may seek "negative opinions about battery life" rather than a general sentiment score for a product. The primary obstacle is that sentiment is highly contextual. A word like "flaw" is a negative indicator in the context of a "presidential election," whereas "reckless" might be the preferred descriptor for a "politician."

Standard information retrieval (IR) models prioritize topical relevance, often ignoring sentiment, while sentiment classifiers frequently overlook topic context. This research proposes a unified generative model to handle both dimensions simultaneously at the sentence level.

Methodology: Generative Language Models

The authors frame the problem using probabilistic generative language models, treating each sentence as a mixture of topic-bearing words, sentiment-bearing words, and a polarity value (positive, negative, or neutral).

Key technical components include:

  • Relevance Models ($R_t$ and $R_s$): The system estimates two linked models—one for the topic and one for the sentiment—calculating the probability of a word appearing given the user's specific information need.
  • Nonparametric Estimation: To capture the dependency between topics and sentiment, a kernel-based density estimator is used. This allows the model to learn which sentiment words co-occur with specific topics in the training data.
  • Cross-Entropy Ranking: Sentences are ranked by calculating the similarity between the sentence's language model and the estimated relevance models using a variation of cross-entropy.
Topic-specific sentiment word distributions showing how sentiment words vary by context.

Experimental Setup and Datasets

The study utilized the MPQA Opinion Corpus (version 1.2), consisting of 11,114 sentences from 535 news articles. The authors derived sentence-level labels from phrase-level annotations and also tested automatic annotation using sentiment word lists (e.g., General Inquirer) and basic negation handling (e.g., "not good" $\rightarrow$ "good_NEG"). This tested whether the model could function effectively without expensive manual labeling.

Results and Performance

The models were evaluated using bpref, a metric robust to incomplete relevance judgments. The Sentiment Language Model (SLM) demonstrated substantial gains over traditional methods:

  • Manual Annotation: The SLM outperformed the standard query likelihood baseline by 76.1% and the conventional relevance model by 32.6%.
  • Automatic Annotation: Even without manual labels, the model performed 67.2% better than the baseline, proving its practical utility.
Performance comparison of various retrieval models across different data subsets.

Experiments confirmed that modeling the dependency between topic and sentiment is significantly superior to treating them as independent variables. The weighting parameter ($\alpha$) allowed for fine-tuning the balance between topical relevance and sentiment accuracy.

Discussion and Future Work

The generative approach offers more flexibility than discriminative classifiers by "imagining" what a relevant sentence should look like for ranking purposes. However, several limitations remain:

  1. Bag-of-Words: The model ignores word order, failing to distinguish between "gone from good to bad" and "gone from bad to good."
  2. Linguistic Structure: It does not account for complex syntax or long-distance dependencies between topics and modifiers.
  3. Polarity Strength: The current ternary scale (-1, 0, 1) could be expanded to a continuous scale to capture the intensity of opinions (e.g., distinguishing "okay" from "spectacular").

Future research will focus on larger-scale datasets to better capture the vast variety of topic-sentiment pairings and incorporating more sophisticated linguistic features.

Summary updated

BibTeX
@inproceedings{eguchi2006sentiment,
  author = {Koji Eguchi and Victor Lavrenko},
  title = {{Sentiment Retrieval using Generative Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2006},
}
Figure