TL;DR
This paper presents SinaAI's approach to SemEval-2023 Task 3, which aims to automatically detect news genre, framing, and persuasion techniques across multiple languages. They achieved strong results by fine-tuning multilingual transformer models like XLM and LaBSE, demonstrating the effectiveness of these models for cross-lingual propaganda detection.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically analyzing news articles for subtle forms of manipulation and bias. Specifically, SemEval-2023 Task 3 focuses on three critical aspects:
- Genre Detection: Distinguishing between different types of news content (e.g., opinion, report, satire).
- News Framing: Identifying how a topic is presented, which aspects are emphasized, and from what perspective.
- Persuasion Technique Identification: Recognizing the rhetorical devices and emotional appeals used to influence readers.
This task is crucial for developing tools that can help users navigate the complex information landscape, understand media bias, and combat the spread of misinformation. The challenge lies in performing these analyses accurately across a diverse set of languages and cultural contexts.
Methods
The SinaAI team employed a strategy centered around fine-tuning pre-trained multilingual transformer language models. Their core approach involved:
- Multilingual Models: They utilized established models like XLM-RoBERTa-Large (XRL), LaBSE (Language-agnostic BERT Sentence Embedding), and mBERT (Multilingual BERT). These models are designed to understand and process text in numerous languages.
- Fine-tuning: The pre-trained models were fine-tuned on the specific sub-tasks of Task 3, adapting them to the nuances of genre, framing, and persuasion detection.
- Ensemble Methods: For sub-tasks 1 and 2, they explored ensemble approaches, combining the predictions of multiple models (e.g., XLM with LaBSE, XLM with mBERT). This often leads to more robust and accurate results than a single model.
- Data Augmentation: To address potential data scarcity for certain languages and sub-tasks, they implemented data augmentation techniques. This involved resampling sentences from original documents to create synthetic training data, helping to improve model generalization.
- System Architecture: They used both single-model architectures (one multilingual model with a classifier head) and ensemble architectures (combining outputs from two models).
Datasets
The primary dataset used was the SemEval-2023 Task 3 dataset. This dataset comprises news articles in nine languages: English, French, Italian, German, Polish, Russian, Georgian, Greek, and Spanish. It includes labeled instances for news framing, genre, and persuasion techniques. Notably, six of these languages had train, development, and test sets, while the remaining three (Georgian, Greek, and Spanish) only had a test set.
Key Insights and Ideas
- Multilingual Embeddings are Powerful: The study strongly reinforces the effectiveness of multilingual sentence embedding models (like LaBSE and XLM) for cross-lingual tasks like propaganda detection. They can transfer knowledge across languages, even for languages with limited training data.
- Ensembling Boosts Performance: Combining multiple well-performing multilingual models through ensembling significantly improved results, particularly for news framing.
- Data Augmentation Helps: Even with large pre-trained models, data augmentation proved beneficial, especially for languages with smaller training sets, by helping models generalize better.
- XLM's Versatility: XLM emerged as a strong performer, often outperforming other models individually and serving as a robust base for ensemble approaches.
Key Results
- Sub-task 1 (Genre Detection): The XLM model with data augmentation achieved the highest Macro-F1 score (0.65) and Micro-F1 score (0.72) on the validation set. SinaAI's models achieved the top score for two languages (Greek and Italian) on the official leaderboard for this sub-task.
- Sub-task 2 (News Framing): The ensemble model combining XLM and LaBSE showed the best performance on the validation set. For the Russian language on the test set, the ensemble of XRL and LaBSE without augmentation outperformed other models.
- Sub-task 3 (Persuasion Technique Identification): The XLM single model achieved a Micro-F1 of 0.60 and a Macro-F1 of 0.13 on the validation set.
Limitations and Future Work
- Limited Hyperparameter Optimization: The authors acknowledge that due to computational resource constraints, they had to limit hyperparameter tuning, which may have impacted the final performance.
- Future Work: The paper suggests that further exploration of hyperparameter optimization and more extensive experimentation with different model combinations and data augmentation strategies could lead to even better results. The effectiveness of multilingual models for propaganda detection is highlighted as a promising area for continued research.
Summary updated