IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English

Tanuj Shekhawat, Manoj Kumar, Udaybhan Rathore, Aditya Joshi, Jasabanta PatroNAACL 2022

TL;DR

This paper presents the IISERB Brains team's approach to identifying intended sarcasm in English text for the SemEval-2022 Task 6. They leveraged BERT-based deep learning models with data augmentation to achieve competitive results across three sub-tasks, highlighting the effectiveness of these architectures for nuanced language understanding.

What Problem Does This Paper Address?

Sarcasm, a form of verbal irony where the literal meaning differs from the intended meaning, is a pervasive element of human communication, especially in online text. Detecting sarcasm is crucial for various Natural Language Processing (NLP) tasks like sentiment analysis, opinion mining, and harassment detection. However, its subtle nature and reliance on context make it a significant challenge for automated systems. SemEval-2022 Task 6 aimed to address this by proposing three sub-tasks focused on identifying sarcasm in English and Arabic text, pushing the boundaries of current sarcasm detection capabilities.

Methods

The IISERB Brains team employed a deep learning framework centered around Transformer-based architectures, specifically utilizing models from the Hugging Face library. For all three sub-tasks, they fine-tuned pre-trained language models like BERT, RoBERTa, XLNet, and DistilBERT.

  • Data Augmentation: To combat the issue of limited and imbalanced datasets, they utilized the NLPAUG library to synthesize additional training data. This involved applying word-replacement techniques to existing sarcastic tweets to create variations.
  • Sub-task A (Sarcasm Detection): A binary classifier was deployed for each Transformer architecture. They also augmented the training set with non-sarcastic versions of sarcastic tweets provided by the organizers.
  • Sub-task B (Ironic Category Identification): This sub-task involved a multi-label classification problem. The team implemented six separate binary classifiers, one for each ironic category, and merged their predictions.
  • Sub-task C (Sarcastic Text Identification): This task was framed as a parallel combination of two Sub-task A models.

Datasets

The primary dataset used was the English dataset provided by the SemEval-2022 Task 6 organizers, containing 3468 tweets (867 sarcastic, 2601 non-sarcastic). To augment this, they incorporated data from:

  • SemEval-2018 Task 3: Training and test data for irony detection.
  • MUSTARD: The textual component of the multi-modal sarcasm detection dataset.
  • FigLang 2020 Sarcasm: Data from a shared task on sarcasm.

After deduplication and augmentation, the final dataset comprised 19986 tweets.

Key Insights and Ideas

  • Transformer Dominance: The paper reinforces the effectiveness of Transformer-based models (like BERT and RoBERTa) as state-of-the-art for complex NLP tasks, including sarcasm detection.
  • Data Augmentation's Value: The use of data augmentation proved crucial in mitigating the challenges posed by small and imbalanced datasets, leading to improved model performance.
  • Task-Specific Architectures: The team demonstrated that tailoring the model architecture to the specific sub-task (e.g., binary classifiers for sub-task B) can yield better results.
  • Author-Annotated Data Uniqueness: The dataset's unique characteristic of having sarcastic labels marked by the author themselves is highlighted as a valuable aspect for capturing genuine sarcastic intent.

Key Results

The IISERB Brains team achieved notable rankings in the competition:

  • Sub-task A: 19th rank out of 43 teams.
  • Sub-task B: 8th rank out of 22 teams.
  • Sub-task C: 13th rank out of 16 teams.

Specific performance metrics are detailed in the tables, with BERT-based models often showing strong performance across the tasks. For instance, in Sub-task C, a BERT-based fine-tuned model achieved an accuracy of 0.62.

Limitations and Future Work

  • Dataset Size: The paper acknowledges that even with augmentation, the dataset size remained relatively small, which can still pose challenges for deep learning models.
  • Hyperparameter Tuning: While efforts were made, the paper notes that the best performing results might differ from those submitted due to the evaluation period constraints.
  • Future Work: The authors suggest that further exploration of different Transformer architectures and more advanced data augmentation techniques could lead to even better performance. The code and resources are made publicly available on GitHub to encourage further research and reproducibility.

Summary updated

BibTeX
@inproceedings{shekhawat2022iiserb,
  author = {Tanuj Shekhawat and Manoj Kumar and Udaybhan Rathore and Aditya Joshi and Jasabanta Patro},
  title = {{IISERB Brains at SemEval-2022 Task 6: A Deep-learning Framework to Identify Intended Sarcasm in English}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure