stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets

Mengfei Yuan, Zhou Mengyuan, Lianxin Jiang, Yang Mo, Xiaofeng ShiNAACL 2022

TL;DR

This paper presents a robust system for detecting sarcasm in English tweets, achieving top performance in the SemEval-2022 Task 6. The authors combined multiple transformer models with external datasets and novel feature engineering techniques, demonstrating the power of ensemble methods and advanced training strategies for this challenging NLP task.

What Problem Does This Paper Address?

Sarcasm is a complex linguistic phenomenon where the intended meaning is the opposite of the literal meaning. Detecting sarcasm in text, especially in the informal and noisy environment of social media like Twitter, is a significant challenge for natural language processing (NLP) systems. Sarcastic tweets often lack explicit negative sentiment markers, making them difficult to distinguish from genuine expressions. This paper tackles the specific problem of identifying intended sarcasm in English tweets, as defined by SemEval-2022 Task 6 (iSarcasmEval), aiming to improve the accuracy and robustness of automated sarcasm detection.

Methods

The authors employed a multi-faceted approach, combining several advanced techniques:

  • Transformer-based Language Models: They utilized powerful pre-trained models like RoBERTa-large, XLM-ROBERTa-large, and DeBERTa-v3-large. DeBERTa-v3-large, in particular, showed superior performance due to its disentangled attention mechanism and enhanced mask decoder.
  • Ensemble Learning (Voting Mechanism): To leverage the strengths of different models and mitigate individual weaknesses, they fused the predictions of 15 individual models using a mixed hard and soft voting strategy. This ensemble approach significantly boosted overall performance.
  • External Datasets: To improve model generalization, they augmented the official training data with three publicly available sarcasm datasets: iSarcasm, Multi-modal Sarcasm data (text only), and the SemEval-2018 Task 3 dataset.
  • Feature Engineering: Beyond raw text, they incorporated several handcrafted features:
  • Emoji: Emojis are recognized as strong indicators of emotion and can signal sarcasm.
  • Parts-of-Speech (POS): Specific POS tags and patterns were mined from sarcastic tweets and their rephrases.
  • Misspelled/Capitalized Words: Non-standard word usage (e.g., "soooo," "LOOVE") and unusual capitalization were identified as potential sarcasm indicators.
  • Transitional Words & Negative Polarity: Words indicating contrast or negation were also used as features.
  • Advanced Training Techniques:
    • Multi-sample Dropout: Applied after the pooling layer of transformer models, this technique accelerated convergence and improved generalization.
  • Contrastive Loss: They incorporated contrastive loss functions (NTXent and SupCon) alongside the standard cross-entropy loss. This helps the model learn better representations by maximizing agreement between different augmented views of the same data.
  • Adversarial Training: Techniques like Projected Gradient Descent (PGD) and Fast Gradient Method (FGM) were used to enhance model robustness against adversarial perturbations.

Datasets

  • Official SemEval-2022 Task 6 (iSarcasmEval) Training Data: Provided English tweets labeled as sarcastic or non-sarcastic.
  • External Datasets:
    • iSarcasm: A public dataset of English tweets with sarcasm labels.
  • Multi-modal Sarcasm data: Used only the text component from this dataset.
  • SemEval-2018 Task 3: A dataset with similar task descriptions and labels.

Key Insights and Ideas

  • Ensembling is Crucial: The paper strongly advocates for ensemble methods, showing that combining predictions from multiple diverse models significantly outperforms any single model.
  • DeBERTa's Strength: DeBERTa-v3-large emerged as a particularly effective base model, likely due to its advanced architecture for capturing nuanced linguistic information.
  • External Data Boosts Generalization: Incorporating data from related sarcasm detection tasks demonstrably improved the model's ability to generalize to unseen data.
  • Feature Engineering Still Matters: Despite the power of pre-trained transformers, carefully engineered features (emojis, POS, word variations) provided valuable complementary information.
  • Contrastive Learning for Nuance: The use of contrastive loss functions proved effective in learning finer-grained representations, crucial for distinguishing subtle sarcasm.

Key Results

The proposed system achieved an F1 score of 0.6052 for the sarcastic class and a Macro F1 score of 0.7675 on the official test set. This performance placed them first among 43 teams in the "SubTask-A-English" leaderboard. The paper also highlights that the multi-sample dropout layer alone provided a significant 4% improvement in F1 score.

Limitations and Future Work

  • Limitations: The paper focuses solely on English tweets for SubTask-A. The raw tweets are acknowledged as noisy, containing user information, URLs, and hashtags, which the pre-processing aimed to mitigate but may still pose challenges.
  • Future Work: While not explicitly detailed, the success of combining various transformer models, external data, and advanced training techniques suggests that further exploration of these areas could lead to even more accurate sarcasm detection systems. The authors also mention that incorporating multi-modal information (e.g., images) could be a future direction, as hinted by the use of the multi-modal dataset.

Summary updated

BibTeX
@inproceedings{yuan2022stce,
  author = {Mengfei Yuan and Zhou Mengyuan and Lianxin Jiang and Yang Mo and Xiaofeng Shi},
  title = {{stce at SemEval-2022 Task 6: Sarcasm Detection in English Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure