MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection

Mamoun Alghaslan, Khaled AlmutairyACL 2024

TL;DR

This paper investigates the effectiveness of fine-tuning large language models (LLMs) like ChatGPT, Llama-3, and Falcon for Arabic stance detection on social media. The study demonstrates that fine-tuning significantly boosts LLM performance, with GPT-3.5-Turbo achieving state-of-the-art results on the MAWQIF dataset, outperforming previous methods and ranking second in the StanceEval2024 competition.

What Problem Does This Paper Address?

In the age of digital communication, social media platforms are crucial for expressing opinions. Stance detection, the task of identifying whether a text expresses a favorable, unfavorable, or neutral viewpoint towards a specific target, is increasingly important for decision-making across various sectors. While stance detection has been explored for English, there's a notable lack of research specifically for Arabic social media, which presents unique linguistic challenges due to its diverse dialects and the prevalence of special characters and emojis. Existing methods often struggle with generalization to new data or targets. This paper aims to bridge this gap by evaluating the performance of fine-tuned LLMs on Arabic stance detection, a task that has seen limited exploration with these advanced models.

Methods

The researchers fine-tuned three prominent Large Language Models (LLMs): ChatGPT-3.5-Turbo, Meta-Llama-3-8B-Instruct, and Falcon-7B-Instruct. They specifically chose the "Instruct" variants of Llama and Falcon due to their pre-trained nature for following instructions. To handle the complexities of Arabic tweets, including special characters and emojis, a pre-processing step using the AraBERT pre-processor was applied before feeding the data to the models. For the local training of Llama-3 and Falcon, they employed Low-Rank Adaptation (LoRA), a technique that significantly reduces the computational resources and memory required for fine-tuning large models by introducing trainable low-rank matrices instead of retraining all parameters. The fine-tuning process involved training for 3 epochs with specific hyperparameters like LoRA rank, dropout rate, batch size, warm-up steps, and learning rate.

Datasets

The primary dataset used for this research is the MAWQIF dataset. This dataset comprises 4,121 tweets written in multi-dialectal Arabic. Each tweet is annotated with stance (Favor, Against, None) towards one of three targets: COVID-19 vaccine, Digital Transformation, and Women Empowerment. Additionally, it includes annotations for sentiment (Positive, Negative, Neutral) and sarcasm (Yes, No). The dataset was split into an 85-15% train-validation split with stratification based on stance. The paper also mentions the StanceEval2024 leaderboard and a blind test set used for evaluation, where their work achieved a second-place ranking.

Key Insights and Ideas

  • Fine-tuning is Crucial for LLMs in Arabic Stance Detection: The study strongly suggests that while LLMs are powerful, fine-tuning them on domain-specific data like Arabic social media significantly enhances their performance for tasks like stance detection.
  • GPT-3.5-Turbo Dominates: Among the evaluated LLMs, GPT-3.5-Turbo demonstrated superior performance, indicating its robust capabilities in understanding and processing multi-dialectal Arabic text for stance detection.
  • LoRA is a Practical Solution for LLM Fine-tuning: The successful application of LoRA for fine-tuning Llama-3 and Falcon highlights its effectiveness in making LLM adaptation more accessible and feasible for researchers with limited computational resources.
  • Arabic Stance Detection Remains Challenging: Despite advancements, the performance variations across models and targets, particularly the struggle with "against" stances for some models, underscore the inherent complexities of Arabic stance detection.

Key Results

  • GPT-3.5-Turbo achieved the highest overall macro F1-score of 82.93% on the validation split. It performed consistently well across all targets: Women Empowerment (90.42%), COVID-19 Vaccine (90.12%), and Digital Transformation (82.86%).
  • Meta-Llama-3-8B-Instruct showed moderate performance with a macro F1-score of 74.33%. It performed well in detecting 'favor' stances but struggled with 'against' stances, especially in the Digital Transformation category.
  • Falcon-7B-Instruct had the lowest performance with a macro F1-score of 40.73%. It exhibited significant struggles with 'against' stances across all categories, impacting its overall effectiveness.
  • The paper's submission to StanceEval2024, using the fine-tuned GPT-3.5-Turbo model on a blind test set, ranked second.

Limitations and Future Work

The authors acknowledge that while fine-tuning LLMs significantly improves stance detection capabilities, challenges remain. The exact reasons for Falcon-7B-Instruct's poor performance are uncertain, suggesting it struggles significantly with classifying 'against' stances. Future work could focus on:

  • Refining Hyperparameters and Instruction Sets: Further optimization of training parameters and prompt engineering could potentially improve the performance of models like Llama-3 and Falcon.
  • Integrating Retrieval-Augmented Generation (RAG): Incorporating RAG to provide a few relevant examples as context within the input query could enhance LLM performance.
  • Exploring More Diverse Datasets: Expanding research to include a wider variety of Arabic social media datasets would improve model robustness and generalization.
  • Investigating Model Architectures: Further research into model architectures and adaptation techniques could lead to more efficient and effective LLMs for Arabic NLP tasks.

Summary updated

BibTeX
@inproceedings{alghaslan2024mgkm,
  author = {Mamoun Alghaslan and Khaled Almutairy},
  title = {{MGKM at StanceEval2024 Fine-Tuning Large Language Models for Arabic Stance Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure