SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection

Youssef Al Hariri, Ibrahim Abu FarhaACL 2024

TL;DR

This paper explores the effectiveness of large language models (LLMs) and BERT-based models for Arabic stance detection, a crucial task for understanding public opinion in the Arab world. They found that while fine-tuned BERT models like MARBERT perform well, newer, larger LLMs, particularly Command-R, achieve superior results with zero-shot prompting, highlighting the growing potential of LLMs for low-resource languages.

What Problem Does This Paper Address?

Stance detection, the task of identifying an author's viewpoint (favor, against, or neutral) towards a specific target, is vital for applications like sentiment analysis and fake news detection. However, applying this to Arabic presents significant challenges due to the language's diverse dialects, complex orthography, and a scarcity of annotated resources compared to English. The StanceEval 2024 shared task aimed to address these issues by focusing on Arabic stance detection. This paper details the SMASH team's participation in this task, evaluating various models to find the most effective approach for this underrepresented language.

Methods

The researchers experimented with two main categories of models:

  • Fine-tuned Models: These models were specifically trained on the Arabic stance detection dataset. They explored several BERT and BART-based architectures, including AraBERT, mBERT, CAMELBERT, and MARBERT. The fine-tuning process involved standard practices like using the AdamW optimizer for a set number of epochs.
  • Zero-shot Models (LLMs): This approach leverages the inherent capabilities of large language models without task-specific fine-tuning. They tested various LLMs (e.g., AceGPT, Gemma, Llama 3, Command-R, WizardLM-2, Mistral, Mixtral) using prompt engineering. Crucially, they found that prompting in English, rather than Arabic, and using Arabic translations for target topics yielded better results, mitigating potential issues with LLMs generating Arabic text.

Datasets

The primary dataset used for this work was Mawqif, a multi-label Arabic dataset for target-specific stance detection. Mawqif contains 4,121 sentences covering three topics: 'COVID-19 vaccine', 'digital transformation', and 'women empowerment'. It includes labels for stance, sentiment, and sarcasm. For their experiments, the authors split the training data into training and validation sets, and the official evaluation used a blind test set provided by the task organizers.

Key Insights and Ideas

  • LLMs Outperform Fine-tuned Models: The study strongly suggests that larger, more recent LLMs, when prompted effectively, can surpass the performance of traditional fine-tuned BERT-based models for Arabic stance detection.
  • Prompt Engineering is Key for LLMs: The success of LLMs in this task was highly dependent on the prompt design. Using English prompts and translating target topics into Arabic proved more effective than direct Arabic prompting, which led to "hallucination" and poor performance.
  • MARBERT's Strength in 'Favor' Class: While Command-R generally performed better, MARBERT showed a particular strength in identifying the 'favor' stance. However, it tended to misclassify 'neutral' cases as 'favor', possibly due to dataset imbalance.
  • Multilingual Models Have Limitations: The authors note that even strong multilingual models might struggle with the nuances of specific languages and dialects, underscoring the need for truly Arabic-specific LLMs.

Key Results

The SMASH team achieved 4th place in the StanceEval 2024 shared task with a macro F1 score of 0.8041 using the Command-R model.

  • Command-R: Achieved the highest overall F1 score of 0.661 and macro F1 score of 0.820 on the test set.
  • MARBERT: Performed best among the BERT-based models, with an F1 score of 0.570 and macro F1 score of 0.770 on the validation set.
  • Confusion Matrix Analysis: Command-R was better at identifying 'against' and 'neutral' classes, while MARBERT excelled at 'favor'.

Limitations and Future Work

The paper acknowledges several limitations:

  • Reliance on Multilingual Models: All models used were either English or multilingual. The authors believe that dedicated Arabic-specific LLMs would offer superior performance and better handling of Arabic dialects and cultural contexts. They attempted to use JAIS but faced hardware limitations.
  • Zero-shot vs. Few-shot Learning: The study primarily used a zero-shot approach. The authors suggest that future work could explore few-shot learning, which has been shown to improve LLM performance by providing in-context examples.

The authors also highlight the importance of developing Arabic-specific LLMs to improve performance on Arabic NLP tasks and suggest exploring few-shot learning techniques for future research.

Summary updated

BibTeX
@inproceedings{hariri2024smash,
  author = {Youssef Al Hariri and Ibrahim Abu Farha},
  title = {{SMASH at StanceEval 2024: Prompt Engineering LLMs for Arabic Stance Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure