TL;DR
This paper presents a system for detecting propaganda techniques in Arabic text, achieving second place in the ArAIEval Shared Task. The authors demonstrate that a multilingual BERT model, when strategically fine-tuned and leveraging outputs from its earlier hidden layers, can effectively identify and locate propaganda, even in a low-resource language like Arabic.
What Problem Does This Paper Address?
Propaganda, a potent tool for manipulating public opinion, is increasingly prevalent online. Detecting and analyzing its various techniques in text is crucial for mitigating its impact. This paper focuses on the specific challenge of identifying propagandistic techniques within Arabic text, a task complicated by the language's rich morphology and syntactic complexity. The ArAIEval Shared Task 1 specifically requires not only classifying the propaganda technique but also pinpointing the exact character spans within the text where these techniques occur. This presents a significant challenge, especially given Arabic's status as a low-resource language for NLP tasks.
Methods
The core of the system is a fine-tuned multilingual BERT (mBERT) model. The authors explored the impact of using representations from different hidden layers of mBERT, inspired by prior work showing that earlier layers can sometimes capture more fundamental linguistic features. For tokenization, they used mBERT's subword tokenizer and truncated/padded sequences to a uniform length of 256 tokens. The task was framed as a sequence tagging problem, specifically a multi-label, multi-class classification task where each of the 23 propaganda techniques was treated as a separate class. The training loss function used was a modified binary cross-entropy that specifically ignored padding tokens. Hyperparameter tuning was performed using the Optuna framework with the NSGA-II algorithm.
Datasets
The primary dataset used is from the ArAIEval 2024 Shared Task 1 (Hasanain et al., 2024b). This dataset consists of Arabic tweets and news articles, provided in JSONLines format and already split into training, validation, and testing sets. Each sample includes the text and the character-level spans of employed propaganda techniques. The dataset contains 23 distinct propaganda techniques, and while the distribution of these techniques is uneven across the corpus, the training and validation sets exhibit similar distributions for each label. The paper also references the ArPro dataset (Hasanain et al., 2024a) as the largest dataset for fine-grained propaganda detection in Arabic, and the ArMeme dataset (Alam et al., 2024) for exploring propaganda in multimodal formats.
Key Insights and Ideas
- Early Layers Matter: The most striking finding is that the earlier hidden layers of mBERT (specifically the 8th layer) consistently outperformed later layers (10th and 12th) in detecting propaganda techniques. This suggests that fundamental linguistic features captured in these lower layers are highly effective for this task.
- Multilingual Potential: Despite Arabic being a low-resource language, mBERT demonstrates significant potential for propaganda detection, even when compared to specialized monolingual models like AraBERTv2.
- Span Detection Nuances: The model showed a tendency to identify shorter spans more effectively than longer ones, likely due to their simpler structure and fewer contextual dependencies.
- Qualitative Observations: The authors noted instances where the model identified plausible propaganda techniques not present in the gold labels, suggesting that human annotation might not always capture all nuances or that the model is picking up on subtle cues.
Key Results
The system achieved second place in the ArAIEval Shared Task.
- The 8th hidden layer of mBERT yielded the best performance among the explored mBERT layers, achieving a Micro F1 score of 0.30 and a Macro F1 score of 0.58 on the test set.
- This performance was notably close to that of AraBERTv2, a specialized monolingual model, indicating the competitive edge of well-tuned multilingual models.
- The model demonstrated higher recall than precision, suggesting it was good at finding potential propaganda but sometimes produced false positives.
- For span categorization, the model achieved a precision, recall, and F1-score of 0.10 for short spans (<= 3 words), compared to 0.02 for long spans (> 3 words).
Limitations and Future Work
- Submission Strategy: The authors submitted the results from the 10th layer of mBERT to meet the submission deadline, even though the 8th layer performed better. This highlights a trade-off between optimal performance and timely submission.
- Class Imbalance: The paper acknowledges the uneven distribution of propaganda techniques in the dataset, which can be a challenge for model training.
- Future Work: The authors suggest future work could focus on further refining these models, addressing class imbalance, and exploring more advanced techniques to enhance detection performance. They also mention the potential of using larger language models like GPT-4 for annotation and detection.
Summary updated