UMUTeam at SemEval-2022 Task 6: Evaluating Transformers for detecting Sarcasm in English and Arabic

José García-Díaz, Camilo Caparros-Laiz, Rafael Valencia-GarcíaNAACL 2022

TL;DR

This paper details UMUTeam's participation in SemEval-2022 Task 6, focusing on sarcasm detection in English and Arabic using transformer models. They explored various linguistic features and embeddings, finding that knowledge integration of these features yielded the best results, though class imbalance remained a significant challenge.

What Problem Does This Paper Address?

Sarcasm, a form of figurative language where words deviate from their literal meaning, poses a significant challenge for Natural Language Processing (NLP) tasks like sentiment analysis and hate speech detection. Its reliance on context, wordplay, and subtle cues makes it difficult for machines to accurately identify. This paper tackles the problem of automatically detecting sarcasm in both English and Arabic text, specifically within the context of the SemEval-2022 iSarcasm shared task, which featured binary and multi-label classification challenges.

Methods

UMUTeam developed separate systems for English and Arabic, employing a pipeline that involved:

  • Preprocessing: Removing hyperlinks, hashtags, quotations, and emojis, and expanding acronyms for English.
  • Feature Extraction: This was a core component, utilizing several types of features:
  • Language-Independent Linguistic Features (LF): Extracted using the UMUTextStats tool, these included Part-of-Speech (POS) tags, and stylometric metrics like Type Token Ratio (TTR) and punctuation usage.
  • Non-contextual Word Embeddings (WE): Pre-trained FastText embeddings were used.
  • Non-contextual Sentence Embeddings (SE): Also derived from FastText.
  • Contextual Sentence Embeddings (BF): Leveraging BERT for English and Arabic BERT for Arabic. These were fine-tuned using Ray-Tune with Tree of Parzen Estimators to optimize hyperparameters.
  • Neural Network Training: A neural network was trained for each feature set, as well as a combined network. Hyperparameter optimization was performed, evaluating shallow and deep learning networks with various activation functions and learning rates. Large batch sizes and dropout were used to combat class imbalance.
  • Feature Integration: Two strategies were explored to combine the outputs of individual models:
  • Ensemble Learning: Hard voting and averaging probabilities.
  • Knowledge Integration: A strategy that combined features from different sources.

Datasets

The primary dataset used was the iSarcasm 2022 dataset, which was annotated by its authors. For each sarcastic text, authors also provided a non-sarcastic rephrased version. The dataset was split into training and validation sets, with a 20% stratified split used for validation to maintain class balance. The dataset included statistics for both binary classification (sarcasm vs. non-sarcasm) and multi-label classification (identifying specific ironic speech mechanisms like irony, satire, understatement, etc.).

Key Insights and Ideas

  • Language-Independent Features are Crucial: The study highlights the importance of language-independent features, especially for Arabic, where they achieved very high performance.
  • Transformers Show Promise, but Need Context: While transformer-based embeddings (BERT) performed well in isolation, their effectiveness was further enhanced when combined with other features.
  • Knowledge Integration Outperforms Ensemble Methods: The paper found that integrating knowledge from various feature sets was more effective than traditional ensemble methods like hard voting or averaging probabilities for combining model predictions.
  • Class Imbalance is a Major Hurdle: The authors repeatedly emphasize that class imbalance significantly impacted performance, particularly for English, leading to models that sometimes performed little better than random classifiers on minority classes.

Key Results

  • Binary Classification (Task 1):
    • English: UMUTeam achieved a F1-score of 17.97% for the sarcastic class, ranking 41st overall. The best result with knowledge integration on the custom validation split was 45.82%.
  • Arabic: A F1-score of 31.75% for the sarcastic class was achieved, ranking 22nd overall. The custom validation split showed much higher performance, with LF achieving a F1-score of 94.07%.
  • Multi-label Classification (Task 2): Results were not officially recorded due to an unknown error. However, analysis on the validation split showed that the models struggled with class imbalance, with performance on minority ironic speech labels being particularly low.

Limitations and Future Work

The authors acknowledge that their primary weakness lies in handling class imbalance. They note that while they employed techniques like weighting classes and using large batch sizes, the performance on minority classes remained poor. For future work, they plan to explore:

  • Data Augmentation: To increase the number of instances for minority classes.
  • Further investigation into Arabic performance: The significant drop in performance from their custom validation set to the official test set for Arabic warrants further analysis, especially given their limited Arabic language expertise.
  • Addressing the unknown error: Investigating and resolving the error that prevented their Task 2 results from being officially recorded.

Summary updated

BibTeX
@inproceedings{garciadiaz2022umuteam,
  author = {José García-Díaz and Camilo Caparros-Laiz and Rafael Valencia-García},
  title = {{UMUTeam at SemEval-2022 Task 6: Evaluating Transformers for detecting Sarcasm in English and Arabic}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure