LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets

Abdessamad Benlahbib, Hamza Alami, Ahmed AlamiNAACL 2022

TL;DR

This paper presents LISACTeam's approach to detecting intended sarcasm in English tweets for SemEval-2022 Task 6. They achieved strong results by fine-tuning the BERTweet transformer model, demonstrating its effectiveness on a challenging dataset with limited and imbalanced training data.

What Problem Does This Paper Address?

Identifying sarcasm in text, particularly on social media platforms like Twitter, is a significant challenge for natural language processing (NLP). Sarcasm often involves saying the opposite of what is meant, making it difficult for automated systems to interpret sentiment accurately. This misinterpretation can negatively impact downstream NLP tasks like sentiment analysis. The SemEval-2022 Task 6 aimed to address this by focusing on "intended sarcasm," distinguishing it from general irony. The task was made particularly difficult by a small training dataset and a significant imbalance between sarcastic and non-sarcastic examples.

Methods

The LISACTeam employed a transformer-based approach, specifically fine-tuning the BERTweet model. BERTweet is a language model pre-trained on a large corpus of English tweets, making it well-suited for this task. The authors describe a preprocessing pipeline that includes removing URLs and converting emojis to their textual equivalents. For Sub-Task A (sarcasm detection), the fine-tuned BERTweet model was used to classify tweets. For Sub-Task C (identifying the sarcastic tweet between a pair of original and rephrased tweets), the model was fed both texts, and the one with the higher probability of being sarcastic was selected.

Datasets

The primary dataset used was from SemEval-2022 Task 6: iSarcasmEval. For the English portion, the training set contained 3468 tweets (867 sarcastic, 2601 non-sarcastic). The test set for Sub-Task A comprised 200 sarcastic and 1200 non-sarcastic tweets. The test set for Sub-Task C included 200 sarcastic tweets along with their non-sarcastic rephrases. The paper notes the imbalanced class distribution (approximately 25% sarcastic) as a key challenge.

Key Insights and Ideas

  • BERTweet's suitability for tweet sarcasm: The BERTweet model, pre-trained on tweets, demonstrated superior performance compared to other transformer models and traditional machine learning approaches, highlighting the benefit of domain-specific pre-training.
  • Preprocessing matters: While the paper experimented with and without preprocessing, the results suggest that removing URLs and converting emojis to text can be beneficial for transformer models.
  • Transformer models outperform traditional methods: Across both Sub-Task A and C, transformer-based models, particularly BERTweet and RoBERTa, consistently achieved higher F1-scores and accuracy than models like LinearSVC, Logistic Regression, and Bi-LSTM/Bi-GRU.
  • Challenging nature of the task: The relatively low F1-scores achieved by even the top systems (around 0.6052 for Sub-Task A) underscore the inherent difficulty of accurately detecting intended sarcasm in short, informal text like tweets.

Key Results

The LISACTeam's BERTweet-based system achieved 8th place on the Sub-Task A leaderboard and 5th place on the Sub-Task C leaderboard. Specifically, their official submission for Sub-Task A (with preprocessing) achieved an F1-sarcastic score of 0.4334, and for Sub-Task C, they achieved an F1-score of 0.7585. The paper also shows that BERTweet base (with preprocessing) outperformed BERTweet base (without preprocessing) for Sub-Task A, suggesting the preprocessing steps were helpful.

Limitations and Future Work

The authors acknowledge the small size and imbalanced nature of the training dataset as significant challenges. They also note that the top-ranked system for English Sub-Task A achieved an F1-score of around 0.6052, indicating room for improvement. Future work suggested includes exploring data augmentation techniques and oversampling methods to address the data limitations and further enhance the performance of sarcasm detection models.

Summary updated

BibTeX
@inproceedings{benlahbib2022lisacteam,
  author = {Abdessamad Benlahbib and Hamza Alami and Ahmed Alami},
  title = {{LISACTeam at SemEval-2022 Task 6: A Transformer based Approach for Intended Sarcasm Detection in English Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure