TL;DR
This paper explores using pre-trained language models (PLMs) as feature extractors for Arabic stance detection, bypassing the computationally expensive fine-tuning process. By leveraging sentence transformers and BERT-based embeddings with aggregation architectures, the authors achieved competitive performance on social media data, demonstrating a resource-efficient approach for NLP tasks.
What Problem Does This Paper Address?
Social media platforms are crucial for public discourse, making stance detection—identifying user opinions on specific issues—a vital NLP task. This is particularly important for understanding public sentiment and informing decision-making on topics like women's empowerment, COVID-19 vaccination, and digital transformation. While pre-trained language models (PLMs) like BERT have significantly boosted NLP performance, fine-tuning them for downstream tasks like stance detection requires substantial computational resources (powerful GPUs) and time. This paper addresses the challenge of achieving high performance in Arabic stance detection without the prohibitive costs associated with fine-tuning PLMs.
Methods
The authors employed three main approaches for stance detection, all centered around using PLMs as feature extractors without fine-tuning:
- Sentence Transformers: They utilized multilingual sentence transformers (e.g.,
paraphrase-multilingual-mpnet-base-v2,LaBSE) to generate sentence embeddings for Arabic tweets. These embeddings were then fed into a logistic regression model for stance prediction. - BERT-based Embeddings: This approach used pre-trained BERT models specifically for Arabic. They extracted sentence embeddings in two ways: by taking the
[CLS]token's output embedding or by averaging all output embeddings from the BERT model. These embeddings were also used with a logistic regression classifier. - Parallel-Sum (P-SUM) on Top of BERT: Inspired by previous work on aspect-based sentiment analysis and multi-task learning, they implemented a P-SUM architecture on top of a frozen BERT model (MARBERT). This architecture was designed to create more powerful sentence embeddings for classification tasks.
The core innovation lies in using these PLMs solely as feature extractors, freezing their weights and adding a lightweight aggregation layer or a simple classifier on top. This contrasts with traditional fine-tuning where the PLM's weights are updated.
Datasets
The primary dataset used in this research is the Mawqif dataset.
- Training Data: 3502 Arabic tweets.
- Testing Data: 619 Arabic tweets. The dataset covers three topics: women empowerment, COVID vaccine, and digital transformation. Tweets are annotated with stance labels ("FAVOR", "AGAINST", "NONE") and also include sentiment and sarcasm annotations, which were not utilized in this specific stance detection task.
For evaluation, the authors created an internal test set by randomly shuffling the available data and picking 500 examples, as the official test set was not released. They also submitted their best-performing model to the StanceEval2024 competition for evaluation on a blind official test set.
Key Insights and Ideas
- Frozen PLMs are Competitive: The study strongly suggests that PLMs, even when used as feature extractors without fine-tuning, can achieve performance comparable to or even surpassing traditional fine-tuning methods, especially when considering the computational savings.
- Arabic-Specific BERTs Outperform Multilingual Transformers: Arabic-specific BERT models (like MARBERT and AraBERT-Twitter) generally performed better than multilingual sentence transformers. This is attributed to the fact that the Arabic BERT models were pre-trained on massive Arabic corpora, making them more attuned to the nuances of the language.
- Aggregation Matters: Adding aggregation architectures (like P-SUM) on top of BERT embeddings can further enhance performance, suggesting that how embeddings are combined and processed is crucial for downstream tasks.
- Training Strategy Impact: For sentence transformers, training on the entire dataset (rather than per target) proved more effective, likely due to increasing the effective training data size and allowing the model to learn common stance expression patterns across different topics.
Key Results
- The authors' submission to the StanceEval2024 competition, which used a majority vote of their top three models (MARBERT AVG, P-SUM-MTL, and AraBERTv0.2-Twitter AVG), ranked fifth out of 16 teams.
- The final F1-score (averaged across "FAVOR" and "AGAINST" categories, denoted as Favg2) on the official test set was 78.62.
- On their internal test set, the best-performing model was P-SUM-MTL with an Favg2 of 76.3, closely followed by MARBERT AVG (76.7) and AraBERTv0.2-Twitter AVG (76.1).
- Sentence transformer models generally achieved lower Favg2 scores (ranging from 51.8 to 70.6) compared to BERT-based approaches.
Limitations and Future Work
- Overfitting: A significant challenge faced was overfitting, primarily attributed to the relatively small size of the training dataset, especially when considering the per-target data distribution.
- Dataset Size: The authors suggest increasing the dataset size by incorporating other published Arabic stance detection datasets (e.g., Mubarak et al., 2022).
- Data Augmentation: Employing data augmentation techniques is proposed to balance label distributions and increase the overall dataset size, which is expected to improve model performance.
- Active Learning: Utilizing active learning to select more informative and diverse data samples is another avenue suggested for enhancing model accuracy and efficiency.
Summary updated