YNU-HPCC at SemEval-2022 Task 6: Transformer-based Model for Intended Sarcasm Detection in English and Arabic

Guangmin Zheng, Jin Wang, Xuejie ZhangNAACL 2022

TL;DR

This paper presents a BERT-based system for detecting intended sarcasm in English and Arabic across three subtasks: binary classification, multi-label classification, and sentence-pair classification. The authors demonstrate that fine-tuning pre-trained BERT models, especially RoBERTa, on multilingual datasets significantly improves performance, particularly for Arabic.

What Problem Does This Paper Address?

Sarcasm is a complex linguistic phenomenon where the literal meaning of words contradicts the intended meaning, often for humorous or critical effect. Detecting sarcasm is a challenging Natural Language Processing (NLP) task because it requires understanding subtle cues, context, and even world knowledge. Existing methods often struggle to capture the deep semantic information embedded in text. This paper tackles the SemEval-2022 Task 6, "iSarcasmEval: Intended Sarcasm Detection In English and Arabic," which aims to advance the state-of-the-art in cross-lingual sarcasm detection across various classification formats.

Methods

The core of the YNU-HPCC system is built upon the BERT (Bidirectional Encoder Representations from Transformers) model, a powerful pre-trained language model known for its ability to capture deep contextual information.

  • Input Layer: Text preprocessing involves removing noise like user mentions and URLs, converting text to lowercase, and crucially, preserving hashtag structures. Tokenization is handled by Hugging Face's tools, which also facilitate word embedding.
  • Context Encoder: The system leverages pre-trained BERT models, including RoBERTa, which is noted for its improved pre-training strategy (dynamic masking, full-sentence training).
  • Fully Connected Layers: These layers are used for downstream classification tasks.
  • Output Layer: The processing here varies by subtask.

For the specific subtasks:

  • Subtask A (Binary Classification): A single BERT model is fine-tuned. The [CLS] token, which aggregates sentence-level semantic information, is fed into a fully connected layer and then a Softmax function to predict the probability of sarcasm.
  • Subtask B (Multi-Label Classification): The Label-Powerset method is employed. To address label imbalance (some sarcasm categories are much rarer than others), the authors introduce a weighted binary cross-entropy loss function. This allows the model to focus more on learning from the scarcer labels.
  • Subtask C (Sentence-Pair Classification): Similar to Subtask A, but the input consists of two sentences separated by a [SEP] token. The [CLS] token from the BERT output is then used for classification.

Datasets

The paper utilizes datasets provided by the SemEval-2022 competition:

  • English Dataset: 3467 data points.
  • Arabic Dataset: 2602 data points.

For Subtask B, only the English dataset was used. For Subtasks A and C, both English and Arabic datasets were used. The authors explicitly state no external corpora were used.

Key Insights and Ideas

  • BERT's Power for Sarcasm: The paper reinforces the effectiveness of transformer-based models like BERT for capturing the deep semantic nuances required for sarcasm detection, outperforming traditional machine learning methods.
  • RoBERTa's Advantage: The RoBERTa-base model consistently yielded better results than the standard BERT-base model, highlighting the benefits of its enhanced pre-training.
  • Multilingual Fine-tuning is Key: Fine-tuning on multilingual datasets significantly boosted performance, especially for the Arabic language. This suggests that shared linguistic features or transfer learning across languages is highly beneficial.
  • Weighted Loss for Imbalance: The use of a weighted binary cross-entropy loss function proved crucial for tackling the label imbalance in multi-label sarcasm classification, allowing the model to learn from less frequent categories.
  • Hashtag Preservation: Retaining hashtag structures during preprocessing was found to improve sarcasm detection accuracy, likely because hashtags often carry significant contextual or ironic meaning.

Key Results

The YNU-HPCC system achieved competitive performance in the SemEval-2022 Task 6:

  • English Rankings: 12/43 for Subtask A, 11/22 for Subtask B.
  • Arabic Rankings: 21/32 for Subtask A, 8/13 for Subtask C.
  • The paper notes that the RoBERTa-base model generally performed better than the BERT-base model.
  • Fine-tuning on multilingual datasets significantly improved results, particularly for Arabic.

Limitations and Future Work

  • Arabic Dataset Size: The authors note that fine-tuning the model solely on the Arabic dataset individually showed worse results. They hypothesize this is due to the limited information extracted from the Arabic dataset alone.
  • Future Work: The authors suggest improving the method of fine-tuning on multiple-language datasets by balancing vectors from different languages to enhance model effectiveness. They also aim to extend the model with improved capabilities to achieve even better outcomes.

Summary updated

BibTeX
@inproceedings{zheng2022ynu,
  author = {Guangmin Zheng and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2022 Task 6: Transformer-based Model for Intended Sarcasm Detection in English and Arabic}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure