akaBERT at SemEval-2022 Task 6: An Ensemble Transformer-based Model for Arabic Sarcasm Detection

Abdulrahman Mohamed Kamr, Ensaf MohamedNAACL 2022

TL;DR

This paper tackles the challenging task of detecting sarcasm in Arabic social media text, a crucial step for accurate sentiment analysis. The authors propose an ensemble model combining three powerful Arabic-specific Transformer models (AraBERT, MARABERT, and AraELECTRA), which, after addressing data imbalance with augmentation, achieves a state-of-the-art F1 score of 93%.

What Problem Does This Paper Address?

Social media is awash with irony and sarcasm, making it difficult to accurately gauge public opinion. Detecting sarcasm is particularly challenging because it often involves saying the opposite of what is meant, relying on implicit cues, cultural context, and evolving slang. This is further complicated in Arabic due to its morphological richness, diverse dialects, and limited readily available resources for NLP tasks. Existing sentiment analysis systems struggle with this nuance, leading to misinterpretations. This paper focuses on SemEval-2022 Task 6, specifically Subtask A, which aims to classify Arabic tweets as either sarcastic or non-sarcastic.

Methods

The core of the proposed solution lies in an ensemble approach leveraging pre-trained Transformer models. The authors first fine-tune three state-of-the-art Arabic Transformer models:

  • AraBERT: A language model inspired by Google's BERT, specifically trained for Arabic text.
  • AraELECTRA: An ELECTRA-based model that uses a generator-discriminator approach for more efficient pre-training, showing strong performance on Arabic tasks.
  • MARABERT: Another BERT-based model trained on a large dataset including a significant portion of Arabic tweets, making it adept at handling colloquial language.

A significant challenge identified was the imbalanced nature of the dataset, with far more non-sarcastic tweets than sarcastic ones. To address this, the researchers employed data augmentation using the NLPAug tool, specifically by inserting or replacing words with their embeddings derived from AraBERT. This helped to balance the classes.

Finally, the predictions from the three fine-tuned models are combined using a hard voting ensemble module. This module selects the majority prediction among the three models to determine the final classification.

Datasets

The primary dataset used in this research is the iSarcasmEval Arabic dataset for SemEval-2022 Task 6, Subtask A. This dataset contains 3,102 tweets, with 2,357 labeled as non-sarcastic and 745 as sarcastic. A separate test set of 1,400 tweets was also provided without labels for evaluation. The dataset includes dialect and sarcasm labels.

Key Insights and Ideas

  • Data Imbalance is a Major Hurdle: The stark imbalance between sarcastic and non-sarcastic tweets significantly impacts model performance. Addressing this through data augmentation is crucial for achieving good results.
  • Ensemble Power: Combining multiple strong Transformer models through hard voting can outperform individual models, especially when those models have complementary strengths.
  • Arabic NLP is Challenging but Evolving: The paper highlights the inherent difficulties of Arabic NLP (dialects, morphology) but also showcases the effectiveness of modern Transformer architectures in overcoming these challenges.
  • Augmentation Boosts Performance: The data augmentation step was critical, leading to a substantial improvement in F1 scores for all models.

Key Results

The paper reports on two experimental setups:

  • Without Data Augmentation: The individual models achieved F1 scores ranging from 69% (AraELECTRA) to 77% (MARABERT), with the ensemble model also at 77%.
  • With Data Augmentation: This is where the significant improvements were seen.
  • AraBERT: 87% F1 score
    • AraELECTRA: 90% F1 score
    • MARABERT: 91% F1 score
    • Ensemble Model: Achieved a remarkable 93% F1 score, outperforming all individual models.

The proposed model ranked sixth in the official SemEval-2022 competition.

Limitations and Future Work

  • Limitations: The paper acknowledges the dataset's relatively small size and imbalance as a primary challenge. While data augmentation helped, it's a common limitation in many NLP tasks. The authors also mention that their model ranked sixth in the competition, implying there's still room for improvement.
  • Future Work: The authors plan to further improve the model's performance by focusing on understanding which features are most essential for prediction and by troubleshooting unexpected model outputs. This suggests a deeper dive into feature importance and error analysis.

Summary updated

BibTeX
@inproceedings{kamr2022akabert,
  author = {Abdulrahman Mohamed Kamr and Ensaf Mohamed},
  title = {{akaBERT at SemEval-2022 Task 6: An Ensemble Transformer-based Model for Arabic Sarcasm Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure