Abstract: Wiltrud Kessler and Hinrich Schütze present a supervised machine learning approach to identify "inconsistent" sentiment words—terms whose dictionary polarity is flipped by their context (e.g., "not good" or "lack of quality"). By utilizing syntactic constructions derived from dependency parses rather than simple word-level features, the authors significantly improve the accuracy of sentence-level sentiment analysis.
The Challenge of Polarity Reversal
Most sentiment analysis systems rely on lexicons of positive and negative words. However, language is context-dependent: a positive word like "worth" becomes negative in the phrase "not worth the price." These are termed inconsistent words. While traditional systems handle simple negation (e.g., "not"), they often struggle with complex "polarity shifters" like "lack of," "hardly," or specific structural reversals. Previous attempts often relied on fixed lists of negators or small word windows, failing to capture reversals where the shifter is several words away from the sentiment word.
Methodology: Syntactic Constructions
The authors propose a two-step approach to capture structural context:
- Inconsistency Classification: Sentences are parsed into dependency trees. A syntactic construction is defined as a path in this tree starting from a sentiment word and ending at another word, recording the parts of speech (POS) and directions (up/down) along the way. A "Bag-of-Constructions" (BoC) classifier is then trained to predict if a word's polarity is flipped based on these structural features.
- Automatic PRC Extraction: The system attempts to automatically discover Polarity Reversing Constructions (PRCs)—specific syntactic patterns that reliably flip sentiment. They use Mutual Information (MI) to identify constructions strongly associated with inconsistent words in a large, automatically labeled corpus.
To train these models without manual labeling, the authors use a heuristic: if a positive dictionary word appears in a sentence labeled as negative, that word is automatically tagged as "inconsistent" for training purposes.
Experimental Setup
The approach was evaluated using several key resources:
- Customer Review Dataset (Hu and Liu, 2004): Used for primary evaluation of sentence-level polarity classification.
- MPQA Subjectivity Lexicon: The base dictionary for sentiment words.
- Camera/Cellphone Review Corpus: A larger dataset of ~58,000 sentences used for automatic PRC extraction.
- Gold PRC Set: A manually annotated set of 70 constructions used to validate the MI-based extraction method.
Key Results
The core insight of this work is that sentiment reversal is a structural phenomenon. Syntactic paths follow the actual logic of the sentence, whereas word-level windows often miss distant negators or include irrelevant noise.
As shown in the results, the Bag-of-Constructions (BoC) approach achieved an accuracy of 80.9%, outperforming the standard dictionary-based baseline (76.1%) and a baseline using a fixed negation list (79.0%). Notably, the BoC method showed a significant jump in Macro F-measure to 77.7, compared to the standard baseline's 70.3. The BoC method also outperformed a "Bag-of-Words" (BoW) approach using the same framework, proving that syntactic features are the primary driver of performance.
Conclusion and Future Work
While the syntactic approach provides a clear performance boost, the authors noted that approximately 50% of "inconsistent" training examples were actually cases where a word was not being used to express sentiment at all (subjectivity issues). Future work aims to integrate subjectivity analysis to filter these cases. Additionally, while the automatically extracted PRCs were noisy, they still provided enough signal to improve classification, suggesting that refining automatic annotation could further enhance the system's robustness across different domains.
Summary updated