PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models

Ishaan Shukla, Ankit Vaidya, Geetanjali KaleACL 2024

TL;DR

This paper presents an ensemble approach using fine-tuned BERT-based models for Arabic stance detection on tweets related to COVID-19 vaccines, digital transformation, and women's empowerment. The authors achieved a strong macro F1 score of 78.02% by combining single-task and multi-task learning strategies, demonstrating the effectiveness of large language models for this challenging task.

What Problem Does This Paper Address?

Stance detection, the task of identifying a speaker's opinion (favor, against, or neutral) towards a topic, is crucial for understanding public discourse. While extensively studied for English, Arabic stance detection remains a less explored area. This is due to the inherent complexities of the Arabic language, including its unique script, rich morphology, and diverse dialects, which make processing and analysis more challenging. Furthermore, the availability of large, high-quality Arabic datasets for stance detection is limited, hindering research progress. This paper tackles this gap by participating in the StanceEval 2024 shared task, focusing on Arabic tweets.

Methods

The core of the approach involves fine-tuning several pre-trained Arabic BERT-based models, including MARBERT, MARBERTv2, AraBERT, and CamelBERT. The researchers explored two main learning paradigms:

  • Single-Task Learning (STL): Each model was fine-tuned independently for the stance detection task.
  • Multi-Task Learning (MTL): Models were trained simultaneously on the primary stance detection task and auxiliary tasks such as sarcasm detection and sentiment detection. The rationale behind MTL is that learning related tasks can improve generalization and performance on the primary task.

A key methodological choice was the use of a focal loss function for MTL. Focal loss is designed to address class imbalance by down-weighting the loss assigned to well-classified examples, allowing the model to focus more on hard-to-classify instances. The authors also employed an ensemble of the best-performing STL and MTL models, using a hard voting mechanism, to maximize overall performance. Pre-processing involved removing links and mentions, and replacing emojis with their descriptions.

Datasets

The primary dataset used for this work is the MAWQIF dataset, specifically curated for the StanceEval 2024 shared task. This dataset contains 3502 Arabic tweets distributed across three topics: COVID-19 Vaccine, Women Empowerment, and Digital Transformation. The dataset is described as multi-label, with annotations for stance, sarcasm, and sentiment. The authors used an 80:20 train-validation split and a separate test set of 619 examples for final evaluation.

Key Insights and Ideas

  • MTL Outperforms STL: The study consistently found that Multi-Task Learning models generally outperformed their Single-Task Learning counterparts, highlighting the benefit of auxiliary tasks in improving stance detection performance.
  • Ensembling is Key: Combining the predictions of multiple well-performing models through ensembling proved to be more effective than relying on any single model, leading to a significant boost in the final macro F1 score.
  • 'None' Class is Challenging: The 'None' stance category consistently showed the worst performance across all topics. The authors hypothesize this is due to a lack of distinct semantic cues or keywords in tweets assigned to this category, making it harder for models to learn discriminative features compared to 'Favor' or 'Against'.
  • Topic-Specific Performance Differences: Despite an even distribution of examples across topics, models performed best on the COVID-19 Vaccine topic, followed by Digital Transformation and then Women Empowerment. This suggests that pre-training data or inherent characteristics of the topics might influence model performance.
  • Overfitting in MTL: While MTL showed promise, the authors observed instances where MTL models tended to overfit the training data more than STL models, suggesting a need for careful task ordering and weighting in MTL setups.

Key Results

The ensemble system achieved a macro F1 score of 78.02% on the StanceEval 2024 shared task. Individual macro F1 scores for the topics were:

  • Women Empowerment: 75.52%
  • COVID-19 Vaccine: 79.85%
  • Digital Transformation: 78.69%

The paper also reports that the best performing models were selected based on epoch-wise validation F1 scores.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Overfitting: The system tends to overfit the provided data. Future work could involve exploring better task ordering and weighting in MTL, or utilizing more data.
  • Low-Resource Settings: The current methods require substantial data. For low-resource scenarios, research into synthetic data generation or more efficient architectures is needed to enable stance detection across multiple languages.
  • Ensembling Improvements: Further exploration of ensembling approaches could lead to even better performance.
  • Data Augmentation: Investigating the use of synthetic data or finding additional real-world data could help mitigate overfitting and improve generalization.
  • Addressing the 'None' Class: Developing specific strategies to better handle the 'None' class, perhaps through specialized feature engineering or model architectures, is a promising avenue.

Summary updated

BibTeX
@inproceedings{shukla2024pict,
  author = {Ishaan Shukla and Ankit Vaidya and Geetanjali Kale},
  title = {{PICT at StanceEval2024: Stance Detection in Arabic using Ensemble of Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure