ANLP RG at StanceEval2024: Comparative Evaluation of Stance, Sentiment and Sarcasm Detection

Mezghani Amal, Rahma Boujelbane, Mariem EllouzeACL 2024

TL;DR

This paper presents a comparative evaluation of fine-tuned BERT models for stance, sentiment, and sarcasm detection in Arabic. The authors found that the AraBERT-twitter model achieved the best performance across all three tasks, highlighting its effectiveness for understanding nuanced language in Arabic dialects.

What Problem Does This Paper Address?

Understanding human language is a complex challenge for AI, especially when dealing with subtle nuances like opinions, emotions, and sarcasm. This paper tackles the problem of accurately detecting these aspects in Arabic text, a language known for its syntactic complexity and diverse dialects. Specifically, the authors aim to:

  • Stance Detection: Identify a writer's viewpoint or stance towards a particular topic. This is crucial for understanding arguments and intentions in various contexts, from political debates to social discussions.
  • Sarcasm Detection: Recognize sarcastic expressions, which often convey the opposite of their literal meaning and pose a significant challenge for sentiment analysis.
  • Sentiment Analysis: Determine the overall attitude or emotion expressed in a text (positive, negative, or neutral).

While progress has been made in these areas, particularly for English, research on Arabic is less extensive. This paper contributes to bridging this gap by evaluating state-of-the-art models on Arabic data.

Methods

The core of this research involves fine-tuning pre-trained BERT-based models on Arabic text for the three detection tasks. The authors explored four specific models:

  1. CAMELBERT-da: Trained on dialectal Arabic data.
  2. MARBERT: Trained on a large corpus of Arabic tweets.
  3. AraBERT: Trained on Modern Standard Arabic (MSA) texts.
  4. AraBERT-twitter: An extended version of AraBERT, specifically trained on a substantial amount of Arabic tweets.

For each model, the authors employed a rigorous hyperparameter tuning process over multiple iterations to maximize performance. This involved adjusting parameters like epochs, batch size, learning rate, and using an AdamW optimizer with a CosineAnnealingLR scheduler.

A key aspect of their methodology is the proposed data preprocessing pipeline, which includes:

  • Removing HTML tags, URLs, and attributions.
  • Using a specialized Arabic tokenizer for meaningful tokenization.
  • Adding special tokens, padding sequences, and creating attention masks.

For the sarcasm detection task, which suffered from class imbalance, the authors implemented data augmentation techniques. This involved substituting words, inserting additional words, and removing certain words to increase the representation of the minority "sarcastic" class.

Datasets

The primary dataset used in this study is MAWQIF (Mawqif), introduced by Alturayeif et al. (2022). This dataset is structured as a multi-label dataset, encompassing 4121 entries across three categories: "COVID-19 vaccination," "digital transformation," and "women's empowerment." Each entry can be tagged with:

  • Stance: "Favor," "Against," or "None."
  • Sentiment: "Positive," "Negative," or "Neutral."
  • Sarcasm: "Sarcastic" or "Non-sarcastic."

The authors note that the MAWQIF dataset comprises various Arabic dialects and subdialects, which adds complexity to automated processing. They also mention using a separate test set, loaded from a CSV file, for final evaluation.

Key Insights and Ideas

  • AraBERT-twitter Dominates: The study consistently found AraBERT-twitter to be the top-performing model across all three tasks (stance, sentiment, and sarcasm detection). This suggests that models specifically trained on a large volume of Arabic tweets are highly effective for capturing nuanced language in this domain.
  • Stance Detection is a Strength: The AraBERT-twitter model achieved a particularly strong macro F1-score of 78.08% for stance detection, indicating its robustness in identifying opinions and viewpoints.
  • Sarcasm Detection Remains Challenging: Despite data augmentation efforts, sarcasm detection proved to be the most challenging task. While the model achieved high accuracy (94.83%), the macro F1-score of 59.51% highlights the difficulty in accurately identifying sarcastic instances, suggesting potential class bias.
  • Dialectal Diversity is a Hurdle: The presence of multiple Arabic dialects within the MAWQIF dataset complicates processing, as evidenced by the dialect distribution analysis which shows Modern Standard Arabic (MSA) as the most dominant, but other dialects also present.
  • Importance of Comprehensive Evaluation: The paper emphasizes that relying solely on accuracy for tasks like sarcasm detection can be misleading due to class imbalance. Using metrics like the F1-score provides a more complete picture of model performance.

Key Results

The AraBERT-twitter model demonstrated the following key performance metrics:

  • Stance Detection: Macro F1-score of 78.08% across all targets.
  • Sarcasm Detection: Macro F1-score of 59.51% across all targets.
  • Sentiment Detection: Macro F1-score of 64.57% across all targets.

The paper also provides per-target F1 scores for each category, showing variations in performance across "Women empowerment," "Covid Vaccine," and "Digital Transformation." For instance, in stance detection, the "Women empowerment" category achieved the highest macro F1-score of 0.8207.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Sarcasm Detection Imbalance: The inherent imbalance in the sarcasm dataset remains a significant challenge, impacting the model's ability to accurately detect the minority class.
  • Class Bias: The high accuracy in sarcasm detection might be attributed to a class bias, where the model favors predicting the majority class.
  • Need for Further Investigation: The results, particularly for sarcasm and sentiment detection, indicate a need for further research and exploration of more advanced techniques.

Future work could involve exploring ensemble models by merging individual models to harness their collective strengths. Additionally, multitask learning is proposed as an avenue to enable models to learn multiple tasks simultaneously, potentially leading to improved performance and knowledge sharing. The authors suggest that a methodical approach with rigorous evaluation is necessary to enhance performance in these complex NLP tasks.

Summary updated

BibTeX
@inproceedings{amal2024anlp,
  author = {Mezghani Amal and Rahma Boujelbane and Mariem Ellouze},
  title = {{ANLP RG at StanceEval2024: Comparative Evaluation of Stance, Sentiment and Sarcasm Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure