Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis

Abdullah I. Alharbi, Mark LeeWANLP 2021

TL;DR

This paper tackles the challenging task of detecting sarcasm and sentiment in Arabic social media text, which is complicated by dialectal variations and the inherent ambiguity of sarcasm. They propose a novel multi-task learning approach that combines powerful contextualized word embeddings (MARBERT) with static embeddings (Ara2Vec and character-level CE) to improve performance on both tasks simultaneously. Their system achieved state-of-the-art results for Arabic sarcasm detection.

What Problem Does This Paper Address?

Understanding human language on social media is a complex endeavor, especially when dealing with nuances like sarcasm. Sarcasm is a form of expression where the intended meaning is the opposite of the literal words used. This makes it particularly tricky for sentiment analysis, as positive-sounding words can actually convey negative sentiment. The paper focuses on Arabic, a language with a rich diversity of dialects that differ significantly from Modern Standard Arabic (MSA) and often lack standardized linguistic rules. This dialectal variation poses a substantial challenge for NLP models. Existing research on Arabic sarcasm detection is limited, and the authors highlight a recent shared task that aimed to address this gap by providing datasets and a benchmark. However, this shared task also presented difficulties, including unbalanced class distributions and the use of tweets from various Arabic dialects.

Methods

The authors propose a multi-task learning (MTL) framework that leverages the strengths of different types of word embeddings. Their approach combines:

  • Static Word Embeddings (SWE): They utilize two types of pre-trained static embeddings:
  • Ara2Vec: A word-level embedding model trained on a large corpus of Arabic text (Wikipedia, Twitter, Common Crawl).
  • Character-level Embeddings (CE): A three-gram character representation model trained on FastText using a dataset of Arabic affect words from various dialects. This is particularly useful for handling the OOV (out-of-vocabulary) problem common in Arabic due to dialectal variations.
  • Contextualized Embeddings (MARBERT): They employ MARBERT, a BERT-based model specifically pre-trained on a massive dataset of Arabic tweets. MARBERT excels at capturing the contextual meaning of words, which is crucial for understanding sarcasm.
  • Multi-Task Learning (MTL): The core of their approach is to train models for both sarcasm detection and sentiment analysis simultaneously. The intuition is that learning these related tasks together can improve generalization and performance on both. They integrate MARBERT's sentence embeddings within an MTL framework.

Their system architecture involves pre-processing tweets, then feeding them into separate pathways for MARBERT and static embeddings. The representations from these pathways are concatenated, and then passed through a deep learning architecture (specifically, a CNN-LSTM network) for the final classification. They also mention using a dropout layer to prevent overfitting.

Datasets

The paper utilizes datasets provided for the WANLP-2021 Workshop shared task on "Sarcasm and Sentiment Detection in Arabic."

  • Training Data: 12,548 tweets.
  • Test Data: 3,000 tweets.

The datasets are annotated for two subtasks:

  • Subtask 1 (Sarcasm Detection): Tweets are labeled as 'True' (sarcastic) or 'False' (not sarcastic).
  • Subtask 2 (Sentiment Analysis): Tweets are labeled as 'NEG' (negative), 'NEU' (neutral), or 'POS' (positive).

The paper notes that the class distributions in the training and test sets are unbalanced, particularly for sarcasm detection.

Key Insights and Ideas

  • The Power of Combination: Combining contextualized embeddings (MARBERT) with static embeddings (especially character-level CE) is more effective than using either alone for Arabic sarcasm and sentiment analysis. This suggests that different embedding types capture complementary information.
  • MTL for Related Tasks: Multi-task learning significantly benefits sarcasm detection, indicating that sentiment analysis provides valuable inductive bias for understanding sarcastic expressions.
  • Character-Level Embeddings for Dialects: Character-level embeddings are crucial for handling the OOV problem and capturing morphological variations in diverse Arabic dialects, outperforming word-level embeddings in some baseline models.
  • Context is King for Sarcasm: The paper explicitly acknowledges the importance of context in determining sarcasm, which is why they opted for contextualized models like MARBERT.

Key Results

The paper reports the following key results:

  • Sarcasm Detection (Subtask 1): Their proposed MTL-CNN-LSTM system achieved state-of-the-art results, obtaining an F1-sarcastic score of 0.623 on the test set and ranking 1st among participants.
  • Sentiment Analysis (Subtask 2): The same MTL-CNN-LSTM system achieved a macro F1-PN score of 0.701 on the test set, ranking 11th out of 27 participants. While not first place, this is still a strong performance.
  • Baseline Comparisons: Character-level embeddings (CE) generally performed better than word-level embeddings (AraVec) when used with a baseline CNN model.

Limitations and Future Work

  • Limitations:
    • The authors acknowledge that their system achieved a respectable but not top-tier result for sentiment analysis, suggesting room for improvement.
  • They observed that some multi-word expressions might need to be segmented and treated differently.
  • The current pre-processing strategy removes all punctuation, but some punctuation marks (like question marks and exclamation marks) could be beneficial for sarcasm detection.
  • Future Work:
    • Investigate the impact of sarcasm on sentiment analysis to further enhance SA performance.
  • Apply advanced pre-processing techniques, particularly for segmenting and treating multi-word expressions.
  • Re-evaluate the pre-processing of punctuation, potentially retaining certain marks that could aid sarcasm detection.
  • Further analysis of errors on the test dataset for sentiment analysis to identify specific areas for improvement.

Summary updated

BibTeX
@inproceedings{alharbi2021multi,
  author = {Abdullah I. Alharbi and Mark Lee},
  title = {{Multi-task Learning Using a Combination of Contextualised and Static Word Embeddings for Arabic Sarcasm Detection and Sentiment Analysis}},
  booktitle = {WANLP},
  year = {2021},
}
Figure