Abstract
Sentiment retrieval represents a significant shift from traditional sentiment classification, focusing on ranking sentences based on both a specific topic and a desired sentiment polarity. This paper introduces a generative language modeling framework designed to address the inherent topic-dependency of sentiment—where the meaning of words like "unpredictable" varies wildly across contexts. By explicitly modeling the relationship between topic-bearing and sentiment-bearing words, the proposed Sentiment Language Model (SLM) achieves up to a 76% improvement over standard retrieval baselines.
Introduction
While sentiment classification (categorizing text as positive or negative) is well-established, sentiment retrieval—finding specific opinions about specific attributes—remains an underserved challenge. A user may seek "negative opinions about battery life" rather than a general sentiment score for a product. The primary obstacle is that sentiment is highly contextual. A word like "flaw" is a negative indicator in the context of a "presidential election," whereas "reckless" might be the preferred descriptor for a "politician."
Standard information retrieval (IR) models prioritize topical relevance, often ignoring sentiment, while sentiment classifiers frequently overlook topic context. This research proposes a unified generative model to handle both dimensions simultaneously at the sentence level.
Methodology: Generative Language Models
The authors frame the problem using probabilistic generative language models, treating each sentence as a mixture of topic-bearing words, sentiment-bearing words, and a polarity value (positive, negative, or neutral).
Key technical components include:
- Relevance Models ($R_t$ and $R_s$): The system estimates two linked models—one for the topic and one for the sentiment—calculating the probability of a word appearing given the user's specific information need.
- Nonparametric Estimation: To capture the dependency between topics and sentiment, a kernel-based density estimator is used. This allows the model to learn which sentiment words co-occur with specific topics in the training data.
- Cross-Entropy Ranking: Sentences are ranked by calculating the similarity between the sentence's language model and the estimated relevance models using a variation of cross-entropy.
Experimental Setup and Datasets
The study utilized the MPQA Opinion Corpus (version 1.2), consisting of 11,114 sentences from 535 news articles. The authors derived sentence-level labels from phrase-level annotations and also tested automatic annotation using sentiment word lists (e.g., General Inquirer) and basic negation handling (e.g., "not good" $\rightarrow$ "good_NEG"). This tested whether the model could function effectively without expensive manual labeling.
Results and Performance
The models were evaluated using bpref, a metric robust to incomplete relevance judgments. The Sentiment Language Model (SLM) demonstrated substantial gains over traditional methods:
- Manual Annotation: The SLM outperformed the standard query likelihood baseline by 76.1% and the conventional relevance model by 32.6%.
- Automatic Annotation: Even without manual labels, the model performed 67.2% better than the baseline, proving its practical utility.
Experiments confirmed that modeling the dependency between topic and sentiment is significantly superior to treating them as independent variables. The weighting parameter ($\alpha$) allowed for fine-tuning the balance between topical relevance and sentiment accuracy.
Discussion and Future Work
The generative approach offers more flexibility than discriminative classifiers by "imagining" what a relevant sentence should look like for ranking purposes. However, several limitations remain:
- Bag-of-Words: The model ignores word order, failing to distinguish between "gone from good to bad" and "gone from bad to good."
- Linguistic Structure: It does not account for complex syntax or long-distance dependencies between topics and modifiers.
- Polarity Strength: The current ternary scale (-1, 0, 1) could be expanded to a continuous scale to capture the intensity of opinions (e.g., distinguishing "okay" from "spectacular").
Future research will focus on larger-scale datasets to better capture the vast variety of topic-sentiment pairings and incorporating more sophisticated linguistic features.
Summary updated