I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques

Adrián Moreno Monterde, Laura Vázquez Ramos, Jacinto Mata, Victoria Pachón ÁlvarezNAACL 2022

TL;DR

This paper tackles the challenging task of detecting "intended sarcasm" in English tweets for SemEval-2022 Task 6. The authors explore both Bayesian networks and a BERT-based deep learning model, ultimately finding that BERT, despite data imbalance issues, offers a more promising approach for this nuanced form of language.

What Problem Does This Paper Address?

Social media has become a primary channel for communication and marketing, but accurately understanding user sentiment is hindered by the prevalence of figurative language, particularly sarcasm. Sarcasm is tricky because it often relies on context, tone, and shared understanding, making it difficult for automated systems to detect. This paper focuses on "intended sarcasm," specifically within the SemEval-2022 Task 6, which involves classifying tweets into various ironic speech categories (sarcasm, irony, satire, understatement, overstatement, rhetorical question). The core challenge lies in the inherent ambiguity of text and the often-unbalanced nature of datasets for such tasks.

Methods

The authors employed two main modeling strategies:

  • Bayesian Networks: This served as their initial approach, utilizing a Naive Bayes classifier. This method assumes attribute independence and calculates probabilities based on individual conditional probabilities of each attribute given the class. They experimented with different text preprocessing pipelines (v1.0 and v3.0) to see their impact.
  • BERT-based Deep Learning Model: This was their final submission and the focus of their best results. They leveraged the BERT-base-uncased model, a powerful transformer-based language model pre-trained on a massive corpus. For this multi-label classification task, they adopted a "binary relevance" strategy, essentially training six separate binary classifiers, one for each potential label. They also experimented with various text processing versions (v1.0 through v5.0), including data augmentation techniques.

Datasets

The primary dataset used was the one provided for SemEval-2022 Task 6, containing 3467 English tweets. However, the authors noted that only the first 867 tweets were useful for their task due to missing labels in the rest. They also encountered significant class imbalance within the dataset.

To address the data scarcity and imbalance, they implemented two data augmentation techniques:

  • Synonym Augmentation: Swapping random words with their synonyms using WordNet to create new training examples.
  • External Database Insertion: Manually adding tweets from an external database, particularly for minority labels like satire, overstatement, and understatement.

Key Insights and Ideas

  • The Challenge of Data Imbalance: The dataset's severe imbalance was a major hurdle. Even with augmentation, achieving high scores on minority classes remained difficult. This highlights a common problem in NLP tasks dealing with nuanced linguistic phenomena.
  • Context is King (Even for Machines): The paper reiterates that understanding sarcasm often requires context beyond the immediate text, a challenge for automated systems. The example tweet in Figure 1, lacking the #sarcasm tag, illustrates this point.
  • Preprocessing Nuance: Surprisingly, the authors found that extensive text preprocessing (like removing characters, mentions, or links) was often "useless" and could even hinder performance. They suggest that subtle linguistic cues, even seemingly insignificant characters, might be crucial for detecting ironic speech.
  • BERT's Power: Despite the challenges, the BERT model demonstrated superior performance, underscoring the effectiveness of large, pre-trained transformer models for complex NLP tasks.

Key Results

  • The Bayesian Network model achieved a macro F1-score of 0.45 with text processing v3.0.
  • The BERT-based model achieved a macro F1-score of 0.75 with approach v3.0 (previous versions + insert data of an external database).
  • Their final submission, using the BERT model with specific preprocessing and augmentation, achieved a macro F1-score of 0.0699.
  • They were ranked 10th out of 22 teams in the SemEval-2022 Task 6 Subtask B.

Limitations and Future Work

  • Data Scarcity and Imbalance: The most significant limitation acknowledged is the small and imbalanced dataset, which directly impacted performance.
  • Label Dependency: The binary relevance strategy used with BERT ignores potential dependencies between labels, which could lead to suboptimal predictions for certain label combinations.
  • Future Work: The authors suggest training with a larger dataset and exploring new preprocessing techniques that can effectively remove irrelevant information while preserving crucial ironic speech patterns. They also propose investigating more sophisticated multi-label classification strategies that account for label dependencies.

Summary updated

BibTeX
@inproceedings{monterde2022i,
  author = {Adrián Moreno Monterde and Laura Vázquez Ramos and Jacinto Mata and Victoria Pachón Álvarez},
  title = {{I2C at SemEval-2022 Task 6: Intended Sarcasm in English using Deep Learning Techniques}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure