Confirming the Non-compositionality of Idioms for Sentiment Analysis

Alyssa Hwang, Christopher HideyACL 2019

TL;DR

This paper investigates whether the sentiment of an idiom can be predicted from the sentiment of its individual words. By comparing component-wise sentiment scores with human-annotated phrase-level sentiment, the authors find no significant correlation, confirming that idioms are indeed non-compositional for sentiment. This has important implications for how sentiment analysis models handle multiword expressions.

What Problem Does This Paper Address?

Natural language is rich with multiword expressions (MWEs), phrases whose meaning isn't simply the sum of their parts. Idioms are a particularly challenging subset of MWEs because their meaning is often completely opaque from the individual words. While it's generally accepted that idioms are non-compositional in terms of their literal meaning, this paper specifically probes whether this non-compositionality extends to sentiment. Traditional sentiment analysis models often treat words individually, which can lead to inaccurate sentiment predictions when encountering idioms. For example, "break a leg" has a positive sentiment, despite the negative connotation of "break." This research aims to empirically confirm whether the sentiment of an idiom is predictable from its constituent words, a crucial step for building more robust sentiment analysis systems.

Methods

The core of this research involves comparing two types of sentiment scores for idioms:

  • Component-wise Sentiment Scores: The authors adapt a method developed by Agarwal et al. (2009) to calculate sentiment for each word within an idiom. This involves:
  • Tokenizing the idiom.
  • Assigning a "pleasantness" score to each word using the Dictionary of Affect in Language (DAL). Synonyms or negated word senses are used if a word isn't directly in the DAL.
  • Z-normalizing scores based on DAL manual data and boosting them by their distance from the mean.
  • A finite state machine handles local negations (e.g., "not good").
  • The final component-wise score for an idiom is the sum of these word scores, normalized by the idiom's length. This method aims to capture the compositional sentiment.
  • Phrase-Level Sentiment Labels: These are the "gold standard" labels provided by the SLIDE (Sentiment Lexicon of IDiomatic Expressions) dataset, which were crowdsourced from multiple annotators. The paper focuses on the "positive percent index" (PPI) from SLIDE, calculated as the percentage of positive votes minus the percentage of negative votes.

The primary analytical technique used is Spearman correlation to measure the relationship between the component-wise sentiment scores and the SLIDE phrase-level sentiment labels.

Datasets

The primary dataset used is the Sentiment Lexicon of IDiomatic Expressions (SLIDE), introduced by Jochim et al. (2018). This lexicon contains approximately 5,000 idiomatic expressions, each annotated by at least ten crowdsourced annotators for sentiment (positive, negative, neutral, or inappropriate). For this study, idioms labeled as positive, negative, or neutral were used, totaling 4,999 idioms. The component-wise sentiment scores were derived using the Dictionary of Affect in Language (DAL) and WordNet.

Key Insights and Ideas

  • Idioms are sentimentally non-compositional: The most significant finding is the lack of a discernible correlation between the sentiment derived from individual words and the overall sentiment of the idiom as judged by human annotators. This empirically supports the hypothesis that idioms' sentiment cannot be reliably predicted from their constituent words.
  • Emotional idioms show interesting patterns: While there's no correlation overall, emotional idioms (positive and negative) tended to have lower mean DAL positive index scores and larger standard deviations compared to neutral idioms. This suggests that emotional idioms might contain emotionally charged words, but their collective sentiment doesn't align with the sum of those individual word sentiments.
  • The "words-with-spaces" approach is insufficient: The study highlights that treating MWEs as simple sequences of words for sentiment analysis can lead to inaccurate results, reinforcing the need for specialized handling of these expressions.

Key Results

  • Spearman Correlation: The Spearman correlation coefficients between component-wise sentiment scores and SLIDE sentiment labels were close to zero for all sentiment classes (positive, negative, neutral). For positive idioms, the correlation was -0.144, which, despite being statistically significant due to the large dataset size, indicates a very weak, inverse relationship. For neutral and negative idioms, the null hypothesis of no correlation could not be rejected.
  • Visualizations: Figures 1-4, which plot component-wise DAL scores against SLIDE positive percent indices, show no clear patterns or clusters, visually reinforcing the lack of correlation.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that the SLIDE positive percent index might not perfectly distinguish between majority negative and neutral votes.
  • The analysis primarily focuses on the "positive percent index" and doesn't deeply explore negative or neutral percent indices, which could offer more nuanced insights.
  • Future Work:
    • Learning domain-specific sentiment for MWEs without manual annotation (e.g., predicting negative sentiment for "high blood pressure" in a medical context).
  • Developing methods to recognize new MWEs as language evolves, especially internet slang.
  • Associating new meanings with existing words and phrases, which is crucial for processing evolving language.
  • The paper strongly advocates for treating MWEs as distinct entities rather than simple word sequences in sentiment analysis.

Summary updated

BibTeX
@inproceedings{hwang2019confirming,
  author = {Alyssa Hwang and Christopher Hidey},
  title = {{Confirming the Non-compositionality of Idioms for Sentiment Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure