Developing Corpora for Sentiment Analysis: The Case of Irony and Senti-TUT (Extended Abstract)

Cristina Bosco, Viviana Patti, Andrea BolioliIJCAI 2015

Abstract

Irony presents a significant challenge for sentiment analysis by acting as a "polarity reverser," where the literal meaning of a statement contradicts the intended sentiment. This paper introduces Senti-TUT, a specialized Italian Twitter corpus designed to study this phenomenon. By comparing human-annotated sentiment with automatic rule-based classifiers, we quantify the distortion caused by irony. Our results show that irony is primarily a "one-way street" from positive to negative, resulting in either full polarity reversal or sentiment attenuation.

Introduction

Sentiment analysis—the automated identification of opinions and emotions in text—struggles with figurative language. Irony is particularly problematic because it involves saying one thing while meaning the opposite. For instance, a politically charged tweet like "Great job on the economy!" may appear positive to a machine but is often biting sarcasm. This research addresses the linguistic complexity of irony, the scarcity of Italian sentiment resources, and the lack of methodological standards for annotating irony in social media.

Methodology and Corpus Development

The authors developed the Senti-TUT project using a three-step framework: collection, annotation, and analysis.

  1. Collection: Tweets were scraped via Web APIs, focusing on Italian political events (e.g., the transition to the Monti government in 2011–2012) to ensure high opinion density.
  2. Annotation: A multi-layered approach was used, including a specific HUM (ironic) tag. Five human annotators achieved a Cohen’s $\kappa$ of 0.65, indicating satisfactory agreement.
  3. Analysis: The "polarity reverser" hypothesis was tested by comparing human labels against Blogmeter, a non-irony-aware automatic classifier.

The project produced two primary datasets:

  • TWNews: 3,288 tweets representing spontaneous political chatter.
  • TWSpino: 1,159 messages from the satirical blog Spinoza.it, providing a concentrated sample of professional irony.

Emotional Profiles of Irony

The study mapped ironic expressions to an ontology based on Ekman’s basic emotions. A key finding was the distinction between "genteel irony" in spontaneous chatter and "bitter irony" in dedicated satire.

Comparison of emotion distributions in ironic tweets from TWNEWS-HUM and TWSPINO.

As shown in Figure 4, spontaneous political tweets (TWNews) often mix joy (21%) and sadness (29%). In contrast, satirical content (TWSpino) is significantly more aggressive, dominated by anger (23%) and disgust (18%).

Impact on Sentiment Classification

Irony acts as a distortion filter. In a subset of ironic tweets, the non-irony-aware classifier failed to match human sentiment in 68.5% of cases. The study distinguishes between two types of shifts:

  • Full Reversal: A complete flip in polarity (e.g., POS $\rightarrow$ NEG).
  • Attenuation: A softening of sentiment (e.g., NEG $\rightarrow$ NONE).
Breakdown of sentiment shifts caused by irony: full reversal vs. attenuation.

The data in Figure 5 confirms that irony is almost exclusively used to make a positive statement reflect a negative sentiment (33.6% of all cases). Furthermore, attenuation (62.7%) is more common than full reversal (37.3%), often shifting strong negative sentiments toward a neutral or objective stance.

Conclusion

The Senti-TUT project demonstrates that irony is a critical hurdle for sentiment analysis, primarily serving as a mechanism for negative critique. Future work will focus on refining annotations to distinguish between sarcasm, parody, and satire, and developing richer semantic models capable of reasoning about complex emotional relationships.

Summary updated

BibTeX
@inproceedings{bosco2015developing,
  author = {Cristina Bosco and Viviana Patti and Andrea Bolioli},
  title = {{Developing Corpora for Sentiment Analysis: The Case of Irony and Senti-TUT (Extended Abstract)}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure