TL;DR
This paper tackles the challenge of determining the sentiment (positive, negative, or neutral) of individual phrases within a larger text, going beyond document-level analysis. They introduce a two-step machine learning approach that first identifies whether a phrase expresses sentiment and then disambiguates its specific polarity, achieving significantly better results than baseline methods.
What Problem Does This Paper Address?
Sentiment analysis, the task of identifying opinions and emotions in text, has traditionally focused on the document level (e.g., classifying an entire review as positive or negative). However, many applications like multi-perspective question answering, summarization, and opinion extraction require a more granular understanding of sentiment at the sentence or even phrase level. The core challenge is that words and phrases can have a different "contextual polarity" than their "prior polarity" (their sentiment out of context). For instance, the word "trust" might have a positive prior polarity, but in the phrase "Environmental Trust," it's used neutrally. Similarly, negation ("not good") or complex phrasing can flip the sentiment of a phrase. This paper aims to automatically identify this contextual polarity for phrases.
Methods
The authors propose a two-step approach:
- Neutral-Polar Classification: This step uses a machine learning classifier (BoosTexter AdaBoost.HM) to determine if a phrase containing a "subjectivity clue" (a word or phrase likely to express sentiment) is neutral or polar (positive, negative, or both). This step employs 28 features, including:
- Word Features: The word token itself, its part-of-speech, its context (previous/next words), and its prior polarity and reliability from a lexicon.
- Modification Features: Binary features indicating relationships between words (e.g., if a word is preceded by an adjective or intensifier, or if it modifies another subjective word).
- Structure Features: Binary features derived from the dependency parse tree, indicating if the word is in a subject position, a copular verb, or part of a passive construction.
- Sentence Features: Counts of subjectivity clues, adjectives, adverbs, pronouns, and modals in the current and surrounding sentences.
- Document Feature: The topic of the document.
- Polarity Classification: For phrases identified as polar in the first step, this second step further disambiguates their polarity into positive, negative, both, or neutral. This step also uses BoosTexter AdaBoost.HM, employing 10 features, including:
- Features from the first step (word token, prior polarity).
- Polarity Features: Binary features capturing negation (local or subject negation), relationships with other polarity words (modifies/modified by polarity, conj polarity), and the presence of general, negative, or positive polarity shifters (words that alter sentiment, like "little" or "deeply").
Datasets
The primary dataset used is the MPQA Opinion Corpus. The authors augment existing annotations in this corpus by manually labeling the contextual polarity of subjective expressions. For their experiments, they divide the corpus into:
- A development set (66 documents, 1,373 sentences, 2,808 subjective expressions) for data exploration and feature development.
- A test set (359 documents, 7,611 sentences, 13,183 subjective expressions) used for 10-fold cross-validation experiments.
They also utilize a prior-polarity subjectivity lexicon compiled from various sources, including the General Inquirer lexicon and manually identified positive/negative adjectives.
Key Insights and Ideas
- Context is King: The paper strongly emphasizes that a word's prior polarity is often insufficient for determining its sentiment in a given context. Negation, complex phrasing, and the role of a word within a sentence can drastically alter its meaning.
- Two-Step Approach is Effective: Separating the task into first identifying if sentiment is expressed (neutral vs. polar) and then what that sentiment is (positive, negative, both) proves to be a robust strategy.
- Rich Feature Sets Matter: The performance gains are directly linked to the complexity and variety of features used, particularly those derived from dependency parsing and specific polarity influencers.
- Prior Polarity is a Weak Baseline: A classifier that simply uses the prior polarity of words performs poorly, highlighting the necessity of contextual analysis.
- "Both" Polarity is Important: The inclusion of a "both" category for expressions with mixed positive and negative sentiment is crucial for accurately capturing nuanced opinions.
Key Results
- Prior-Polarity Classifier Baseline: Achieved only 48% accuracy, with significant errors stemming from non-neutral prior polarity words appearing in neutral contexts.
- Neutral-Polar Classification (Step 1): The 28-feature classifier achieved an accuracy of 75.9%, with a polar F-measure of 63.4 and a neutral F-measure of 82.1, significantly outperforming simpler baselines.
- Polarity Classification (Step 2): The 10-feature classifier achieved an accuracy of 65.7%, a positive F-measure of 65.1, and a negative F-measure of 77.2, again significantly outperforming baselines.
- Inter-Annotator Agreement: The manual annotation study showed high agreement (82% overall, 90% when annotators were certain), with a Kappa value of 0.72, indicating the reliability of their annotation scheme.
Limitations and Future Work
- Expression Boundaries: The system does not explicitly identify the boundaries of sentiment expressions; it labels instances of clues. The authors suggest that identifying expression boundaries could improve performance and is a promising avenue for future research.
- Feature Engineering: While the feature sets are extensive, there's always room for further exploration and refinement.
- Broader Context: The current system relies on sentence and document-level features. Incorporating broader discourse context or world knowledge could potentially enhance accuracy.
- "Both" Category Performance: While the "both" category is included, its performance metrics are not as detailed as positive/negative/neutral, suggesting potential areas for improvement.
Summary updated