dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers

Mohamed Lichouri, Khaled Lounnas, Ouaras Rafik, Mohamed ABi, Anis GuechtouliACL 2024

TL;DR

This paper introduces dzStance, a system for Arabic stance detection that leverages Sentence Transformers to outperform traditional TF-IDF methods. The system achieved competitive results in the StanceEval 2024 shared task, demonstrating the effectiveness of advanced embedding techniques for understanding nuanced opinions in Arabic text.

What Problem Does This Paper Address?

Accurately determining public opinion and attitudes on critical societal issues is a growing challenge, especially with the explosion of user-generated content online. This is particularly complex for Arabic, a language rich in morphology, dialects, and intricate syntax. The StanceEval 2024 shared task aimed to address this by focusing on Arabic stance detection for topics like COVID-19 vaccines, digital transformation, and women's empowerment. Traditional methods like TF-IDF struggle to capture the deeper semantic relationships and contextual nuances required for this task.

Methods

The dzStance system explored two main approaches:

  • Traditional TF-IDF: This involved using Term Frequency-Inverse Document Frequency (TF-IDF) features, combined using scikit-learn's FeatureUnion. They experimented with various n-gram ranges (from unigrams up to 10-grams) and weighting schemes. The best performance was achieved with n-grams up to six words, combined with a Linear Support Vector Classifier (LSVC).
  • Sentence Transformers: This approach utilized pre-trained language models, specifically the xlm-r-bert-base-nli-stsb-mean-tokens model, to generate dense sentence embeddings. These embeddings, which capture rich semantic representations, were then fed into a Logistic Regression (LR) classifier. This method also included text preprocessing steps like normalization and emoji replacement.

Datasets

The primary dataset used in this study was the Mawqif dataset (Alturayeif et al., 2022), which was also used for the StanceEval 2024 shared task. This dataset contains over 4,000 annotated Arabic text samples covering stances (favorable, opposing, neutral) on COVID-19 vaccines, digital transformation, and women's empowerment. The paper notes class imbalances within the dataset, particularly for neutral stances.

Key Insights and Ideas

  • Sentence Transformers Shine: The most significant insight is the clear superiority of Sentence Transformers over traditional TF-IDF methods for Arabic stance detection. This highlights the power of deep learning models in capturing nuanced semantic meaning.
  • N-gram Sweet Spot: For TF-IDF, there's an optimal range for n-grams. While longer n-grams can capture more context, going beyond six words started to introduce noise and degrade performance.
  • Hybrid Approaches Hold Promise: The paper suggests that combining the strengths of TF-IDF features with advanced embedding models could lead to even greater improvements in robustness and accuracy.
  • Arabic NLP Complexity: The study underscores the ongoing challenges and importance of developing sophisticated NLP tools for Arabic, given its linguistic richness and variability.

Key Results

The dzStance system achieved competitive results in the StanceEval 2024 shared task:

  • Overall: The system ranked 13th out of 15 teams with an overall average F1-score of 71.77%.
  • Topic-Specific Performance:
    • Women Empowerment: 13th position with an F1-score of 74.91%.
  • COVID-19 Vaccine: 10th position with an F1-score of 73.43%.
  • Digital Transformation: 12th position with an F1-score of 66.97%.
  • Model Comparison: The Sentence Transformer approach with Logistic Regression achieved an F1-score of 68.48%, surpassing the best TF-IDF-based LSVC experiment (which reached 66.20% with six-grams).

Limitations and Future Work

  • Limitations: The paper acknowledges that while the Logistic Regression model with Sentence Transformers performed well, further investigation into why it outperformed traditional methods like LSVC could be beneficial. The presence of class imbalances in the Mawqif dataset is also noted as a challenge.
  • Future Work: The authors suggest exploring further fine-tuning of pre-trained models, investigating hybrid approaches that combine TF-IDF and embedding-based methods, and conducting broader evaluations on varied datasets. They also emphasize making their code and methodologies openly accessible to foster reproducibility and advance the field.

Summary updated

BibTeX
@inproceedings{lichouri2024dzstance,
  author = {Mohamed Lichouri and Khaled Lounnas and Ouaras Rafik and Mohamed ABi and Anis Guechtouli},
  title = {{dzStance at StanceEval2024: Arabic Stance Detection based on Sentence Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure