CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic

Abdelkader El Mahdaouy, Abdellah El Mekki, Kabil Essefar, Abderrahman Skiredj, Ismail BerradaNAACL 2022

TL;DR

This paper presents a transformer-based system for detecting intended sarcasm in English and Arabic tweets, achieving top performance on the Arabic sub-task A. The authors explore various deep learning models and training strategies, finding that including non-sarcastic rephrases of tweets significantly boosts performance, especially for Arabic.

What Problem Does This Paper Address?

Sarcasm, a form of figurative language where the intended meaning differs from the literal, poses a significant challenge for Natural Language Processing (NLP) tasks like sentiment analysis and opinion mining. Accurately detecting sarcasm is crucial for understanding nuanced human communication. While previous research has primarily focused on English, there's a growing need for effective sarcasm detection in other languages, particularly Arabic, which has diverse dialects. The iSarcasmEval shared task was introduced to address this gap, specifically focusing on "intended sarcasm" – the author's actual intent rather than a reader's interpretation. A key challenge is the potential sub-optimality of distant supervision methods (like hashtags) for labeling intended sarcasm, and the limited availability of labeled data, especially for Arabic.

Methods

The CS-UM6P team developed a system leveraging three transformer-based deep learning models. All models utilize pre-trained language models (PLMs): Twitter-XLM-Roberta-base for English and MARBERT for Arabic. These PLMs are fine-tuned using a single attention layer applied to the contextualized word embeddings. A common classifier architecture, comprising one hidden and one classification layer, is used.

The three models differ in their approach to the classification task:

  • Model 1: A binary classifier for sub-task A.
  • Model 2: A multi-class classifier for sub-task A.
  • Model 3: A GAN-BERT-inspired model. This model uses a conditional generator to create fake embeddings similar to real input embeddings, aiming to improve performance on smaller datasets. The generator is trained to fool a discriminator, which also classifies real texts.

The authors also investigated the impact of including non-sarcastic rephrases of tweets in the training data and explored various loss functions (BCE, Weighted BCE, Focal Loss) to handle class imbalance. For their official submissions, they used a hard voting ensemble of their trained models.

Datasets

The paper utilizes the datasets provided for the iSarcasmEval shared task (Abu Farha et al., 2022). These datasets are collected from Twitter and labeled for intended sarcasm by the authors themselves, who also provided non-sarcastic rephrases.

  • English: Training data has 3,468 samples (867 sarcastic).
  • Arabic: Training data has 3,102 samples (745 sarcastic).

The task is divided into three sub-tasks:

  • Sub-task A (Binary Classification): Detects if a tweet is sarcastic or not (Arabic and English).
  • Sub-task B (Multi-label Classification): Categorizes tweets into specific types of ironic speech (English only).
  • Sub-task C (Pairwise Classification): Identifies the sarcastic tweet from a pair of tweets with the same intended meaning (Arabic and English).

Key Insights and Ideas

  • Value of Non-Sarcastic Rephrases: Including non-sarcastic rephrases of sarcastic tweets in the training data significantly improves performance, particularly for the Arabic language. This suggests that providing explicit contrastive examples is highly beneficial for learning intended sarcasm.
  • Model Simplicity vs. Complexity: Surprisingly, simpler models (Model 1 and Model 2) outperformed the more complex GAN-based Model 3 on sub-task A. This might indicate that the GAN approach, while promising for general domain adaptation, may not be optimal for this specific task or dataset size.
  • Loss Function Impact: The choice of loss function plays a role, with Focal Loss and BCE showing competitive results for different models and tasks.
  • Ensembling Drawbacks: Contrary to common practice, ensembling models trained with and without rephrases negatively impacted the official submission results for sub-task C. This suggests that the models might have learned different aspects, and their combination didn't yield a synergistic improvement.

Key Results

  • Sub-task A (Arabic): The system achieved the best performance on sub-task A for Arabic. The best-obtained accuracy scores using non-sarcastic rephrases were 0.85 (Model 2 with FL) and 0.71 (Model 1 with BCE) for Arabic and English respectively.
  • Sub-task A (English): The system ranked 15th for English on sub-task A.
  • Sub-task B: The system ranked second on sub-task B.
  • Sub-task C: The system ranked 7th on Arabic and 11th on English for sub-task C.
  • The best-obtained accuracy scores for sub-task C using non-sarcastic rephrases were 0.85 and 0.71 for Arabic and English respectively, outperforming the official submission scores of 0.78 and 0.69.

Limitations and Future Work

  • Ensembling Impact: The paper notes that ensembling models trained with and without rephrase data negatively impacted the official results for sub-task C. This suggests a need for more sophisticated ensembling strategies or further investigation into why this occurred.
  • Hyperparameter Tuning for FL: For sub-task B, the authors mention that the FL and W. BCE loss functions did not improve results, possibly due to not tuning the hyper-parameters for the Focal Loss.
  • Model 3 Performance: The GAN-based model (Model 3) did not perform as well as simpler models on sub-task A, suggesting potential areas for improvement or alternative GAN architectures.
  • Future Work (Implied): Further exploration into more effective ensembling techniques, deeper hyperparameter optimization for various loss functions, and potentially more advanced GAN architectures could be avenues for future research. The success with non-sarcastic rephrases also points to the value of contrastive learning in this domain.

Summary updated

BibTeX
@inproceedings{mahdaouy2022cs,
  author = {Abdelkader El Mahdaouy and Abdellah El Mekki and Kabil Essefar and Abderrahman Skiredj and Ismail Berrada},
  title = {{CS-UM6P at SemEval-2022 Task 6: Transformer-based Models for Intended Sarcasm Detection in English and Arabic}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure