TL;DR
This paper introduces SenSim, a novel word similarity metric that incorporates sentiment analysis. By considering the positive or negative sentiment of words, SenSim demonstrates improved correlation with human judgment and enhances performance in downstream NLP tasks like sentiment classification.
What Problem Does This Paper Address?
Traditional word similarity metrics, often based on semantic relationships within resources like WordNet, primarily focus on meaning. However, this approach can fall short when sentiment plays a crucial role in determining word interchangeability. For instance, while "refuge" and "asylum" are semantically similar, "mad house" is not a suitable replacement for "refuge" due to its negative connotation. This paper addresses the gap by proposing a metric that leverages sentiment information to compute word similarity, hypothesizing that sentiment is a crucial, yet often overlooked, factor in accurately measuring semantic relatedness.
Methods
The core of SenSim lies in representing words as vectors derived from their WordNet glosses. Each dimension of this "gloss vector" captures the sentiment score of content words within the gloss. To enrich these vectors and mitigate sparsity, the authors augment the original gloss with the glosses of related synsets, carefully selecting WordNet relations based on Part-of-Speech (POS) categories. Sentiment scores are obtained from SentiWordNet, which provides positive, negative, and objective scores for synsets. The authors explore four scoring functions (Sentiment Difference, Sentiment Threshold Difference, Sentiment Max, and Sentiment Threshold Max) to derive a single sentiment value from these scores. Finally, the similarity between two words is computed using the cosine similarity of their augmented gloss vectors.
Datasets
- WordNet: Used as the primary semantic resource for word senses and glosses.
- SentiWordNet 1.0: Utilized as an external sentiment lexicon to derive sentiment scores for synsets.
- Custom-annotated dataset: A new dataset of 48 random word pairs was created for intrinsic evaluation. These pairs were sense-disambiguated using WordNet and annotated by human annotators on a 1-5 scale, both based on meaning alone and on a combination of meaning and sentiment.
- Dataset for synset replacement: Not explicitly detailed, but implied to be a dataset used in prior work for supervised sentiment classification, likely involving documents with associated sentiment labels.
Key Insights and Ideas
- Sentiment is crucial for similarity: The paper strongly supports the hypothesis that incorporating sentiment information significantly improves word similarity measurement, leading to better agreement with human annotators.
- Augmenting gloss vectors is effective: Expanding gloss vectors with related synset information helps to create richer representations and reduces sparsity, leading to more robust similarity calculations.
- POS-specific sentiment roles: Sentiment appears to play a more significant role in similarity for verbs and adjectives compared to nouns, suggesting that the impact of sentiment can vary across word categories.
- SenSim outperforms existing metrics: SenSim, particularly its TD and TM variants, demonstrates superior correlation with human judgments and better performance in extrinsic evaluations compared to established WordNet similarity metrics like Lesk, LIN, and LCH.
Key Results
- Intrinsic Evaluation: The annotation strategy combining meaning and sentiment yielded a higher inter-annotator correlation (0.799) compared to meaning alone (0.768). SenSim variants showed better correlation with the gold standard data than other metrics, with SenSim (TD) achieving the highest correlation (0.50) among all tested metrics.
- Extrinsic Evaluation (Synset Replacement): SenSim (TM) achieved the highest classification accuracy (90.17%) in a document-level sentiment classification task using a synset replacement strategy. This indicates SenSim's effectiveness in mitigating the "unseen feature problem" in supervised learning. SenSim variants generally improved classification accuracy over the baseline and other WordNet metrics.
Limitations and Future Work
- Coverage of Metrics: The paper notes that some existing metrics (e.g., LIN, LCH) could not handle all POS categories, limiting their applicability. SenSim, however, demonstrated better coverage.
- Sentiment Role Variation: The authors suggest that sentiment might be more beneficial for certain POS categories (like verbs and adjectives) than others (like nouns), implying that future work could explore POS-specific sentiment integration.
- Future Work: The paper implicitly suggests further exploration of different scoring functions and WordNet relations for gloss vector augmentation. The success in the synset replacement task also opens avenues for applying SenSim in other NLP tasks that benefit from sentiment-aware similarity.
Summary updated