TL;DR
This paper tackles the challenging task of detecting propagandistic techniques in Arabic text, particularly tweets and news articles. The authors achieve strong results by fine-tuning an Arabic BERT model for sequence tagging, finding that focusing on the first token of a word and incorporating genre information significantly boosts performance.
What Problem Does This Paper Address?
The proliferation of propaganda online poses a significant threat to informed public discourse. Identifying and countering these techniques is crucial, but doing so in Arabic presents unique challenges due to the language's complex morphology and the prevalence of diverse dialects. Existing methods often struggle with the nuances of Arabic text, leading to the need for specialized approaches. This paper addresses the specific problem of detecting propagandistic techniques within Arabic text snippets, as defined by the ArAIEval shared task, aiming to improve accuracy and robustness.
Methods
The core of the system is built upon the AraBERT v2 pre-trained model, a BERT variant specifically designed for Arabic. This model is used to generate contextual embeddings for each token in the input text. The authors then employ a neural network classifier for sequence tagging, where the goal is to identify the spans of text that constitute specific propaganda techniques.
Several sequence tagging prediction approaches were explored:
- Token-Level Training and Prediction: The model is trained on token-level labels, with predictions made for each token. A limitation here is that predicted spans can sometimes fall in the middle of words.
- Token-Level Training with Word-Level Prediction: To mitigate the token-level span issue, predictions are aggregated at the word level. Two strategies were tested:
- Majority Label Assignment: The word receives the label that appears most frequently among its constituent tokens.
- First Token Label Assignment: The word is assigned the label of its first token. This proved to be the most effective word-level aggregation strategy.
- Word-Level Training and Prediction: The model is trained directly on word-level labels, achieved by max-pooling token embeddings to create word embeddings.
A key innovation is the concatenation of the [CLS] token embedding with individual token embeddings. This allows the model to leverage both global sequence representation and local token context. Additionally, genre encoding (tweet vs. news paragraph) was incorporated as a one-hot vector feature, concatenated with the token embeddings, to provide the model with genre-specific context.
Datasets
The dataset used in this work is from the ArAIEval 2024 task 1. This dataset comprises multi-genre text snippets, including tweets and news paragraphs, annotated with propaganda techniques and their corresponding spans. The paper provides basic statistics on the dataset distribution across training, development, and test sets.
Key Insights and Ideas
- The "First Token" Rule: For Arabic, the first token of a word often carries its primary meaning. Leveraging this for word-level prediction significantly improved performance compared to majority voting or word-level training alone. This highlights the importance of linguistic nuances in Arabic for NLP tasks.
- Genre Matters: Explicitly informing the model about the text's genre (tweet vs. news) provided a substantial performance boost across most sequence tagging approaches. This suggests that propaganda techniques manifest differently in different media, and models benefit from this contextual information.
- Simplicity in Architecture: Surprisingly, the best-performing classifier architecture had no hidden layers. This indicates that the powerful representations from AraBERT, combined with the chosen sequence tagging strategy and genre encoding, were sufficient without complex post-processing layers.
- [CLS] Token Benefits: Concatenating the [CLS] token embedding with token embeddings yielded better results than ignoring it, demonstrating the value of capturing the overall sequence representation alongside token-specific context.
Key Results
The system achieved a Micro F1 score of 25.41, placing 4th on the ArAIEval leaderboard. Subsequent post-submission improvements further raised the score to 26.68.
The Word-to-Word (First-Label) approach combined with genre encoding achieved the highest Micro F1 score of 26.68. This approach significantly outperformed other methods, including token-to-token and token-to-word (majority-label) strategies. Incorporating genre information consistently improved performance by several percentage points across different approaches, except for the word-to-word approach where it was less impactful.
Limitations and Future Work
- Convergence Time: While genre encoding improved performance, it increased the convergence time of the model, extending it to around 10 epochs compared to the usual 3.
- Manual Correction: Despite automated preprocessing, some manual correction of substring search failures was necessary due to Unicode handling discrepancies.
- Future Directions: The authors suggest continued exploration of integrating additional contextual information and refining techniques to enhance propaganda detection across various languages and genres. They also emphasize the importance of methodological choices in improving model accuracy.
Summary updated