GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification

Diksha Krishnan, Jerin Mahibha C, Thenmozhi DurairajNAACL 2022

TL;DR

This paper presents a system for detecting sarcasm in English text, tackling three subtasks: binary sarcasm detection, multi-label irony type classification, and identifying sarcastic phrases from rephrased versions. The authors leverage ELMo embeddings with Convolutional Neural Networks for binary classification and Gaussian Naive Bayes with TF-IDF for multi-label classification, achieving moderate results on the SemEval-2022 iSarcasmEval task.

What Problem Does This Paper Address?

Sarcasm, a form of irony where words convey the opposite of their literal meaning, is prevalent in online communication. Its detection is crucial for accurate sentiment analysis, opinion mining, and understanding user intent. However, sarcasm is inherently contextual and often relies on shared knowledge between speaker and audience, making it a challenging NLP task. This paper addresses the need for robust sarcasm detection systems by participating in SemEval-2022's iSarcasmEval challenge, which specifically targets sarcasm and irony in English and Arabic. The challenge is divided into subtasks that require distinguishing sarcastic from non-sarcastic text, classifying different types of irony, and identifying the original sarcastic phrase when presented with a non-sarcastic rephrase.

Methods

The proposed system employs a hybrid approach combining deep learning and traditional machine learning techniques.

  • Subtasks A (Sarcastic vs. Non-Sarcastic) and C (Sarcastic vs. Rephrase): These binary classification tasks are tackled using a Convolutional Neural Network (CNN) architecture. The core of this approach is the use of ELMo (Embeddings from Language Models) for contextual word embeddings. ELMo generates deep, contextualized word representations by using bidirectional LSTMs, capturing both syntax and semantics of words within their linguistic context. The ELMo embeddings are fed into an embedding layer, followed by two dense layers, with the final layer using a softmax activation for binary classification.
  • Subtask B (Different types of irony): This multi-label classification task is addressed using a Gaussian Naive Bayes classifier. Instead of deep contextual embeddings, this approach relies on TF-IDF (Term Frequency-Inverse Document Frequency) vectors extracted from the text. TF-IDF is a statistical measure used to evaluate the importance of a word in a document relative to a collection of documents.

Datasets

The study utilizes the training and testing datasets provided by the organizers of the SemEval-2022 Task 6: iSarcasmEval.

  • Training Data: For Subtask A, the training data consisted of 867 sarcastic and 2601 non-sarcastic instances. For Subtask B, the data included instances labeled with six irony types: sarcasm, irony, satire, understatement, overstatement, and rhetorical question. Subtask C involved pairs of sarcastic phrases and their non-sarcastic rephrases.
  • Evaluation Data: A portion of the training data was used for evaluation. Specifically, 80% of the training instances were used for training, and the remaining 20% were used for evaluation. The test data, comprising 1400 instances, was used for final predictions.

Key Insights and Ideas

  • ELMo's Contextual Power: The use of ELMo embeddings for Subtasks A and C highlights the importance of capturing nuanced word meanings based on their surrounding context for effective sarcasm detection. This is particularly relevant for sarcasm, where literal meaning is often subverted.
  • TF-IDF's Role in Multi-Label Classification: The choice of TF-IDF with Gaussian Naive Bayes for Subtask B suggests that for distinguishing between multiple specific irony types, simpler statistical features might offer a viable, albeit less complex, alternative to deep contextual embeddings.
  • Data Imbalance is a Challenge: The significant imbalance in the training data for Subtask A (867 sarcastic vs. 2601 non-sarcastic) is acknowledged as a factor that can affect model performance, underscoring the ongoing challenge of handling imbalanced datasets in NLP.

Key Results

The proposed models achieved the following macro-F1 scores on the SemEval-2022 iSarcasmEval task:

  • Subtask A (Sarcastic vs. Non-Sarcastic): Macro-F1 score of 0.2012. The team ranked 36th on the leaderboard for this subtask.
  • Subtask B (Different types of irony): Macro-F1 score of 0.0387. The team ranked 19th on the leaderboard.
  • Subtask C (Sarcastic vs. Rephrase): The evaluation metric was accuracy, with the model achieving 0.34. The F1 score was 0.2794. The team ranked 15th on the leaderboard for this subtask.

The authors note that the performance of their models was "not up to the mark" and suggest that transformer models generally provide better results for these tasks.

Limitations and Future Work

  • Performance Gap: The authors explicitly state that their model's performance was not as high as desired, particularly when compared to transformer-based models, which are noted as providing better results. This suggests a limitation in the chosen ELMo and Gaussian Naive Bayes approaches for this specific task.
  • Future Directions: The paper suggests that hybrid approaches, combining various machine learning and deep learning models, could improve sarcasm detection. Additionally, the authors point to multimodal sarcasm detection (integrating text with images or other modalities) as a promising research area, as sarcasm can often be conveyed through non-textual cues like intonation or facial expressions. They also emphasize the need for datasets that better capture contextual information for more effective sarcasm detection.

Summary updated

BibTeX
@inproceedings{krishnan2022getsmartmsec,
  author = {Diksha Krishnan and Jerin Mahibha C and Thenmozhi Durairaj},
  title = {{GetSmartMSEC at SemEval-2022 Task 6: Sarcasm Detection using Contextual Word Embedding with Gaussian model for Irony Type Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure