TL;DR
This paper explores whether knowledge transfer from media bias detection can improve the performance of detecting persuasive techniques in online news. While beneficial for genre detection, the authors found that this transfer learning approach yielded only moderate results for persuasive technique detection, particularly struggling with less common techniques.
What Problem Does This Paper Address?
In today's digital landscape, misinformation and propaganda are rampant. A crucial step in combating these issues is the ability to automatically detect and analyze persuasive techniques used in online news articles. This is a challenging task, especially in a multilingual context, as persuasive language can be subtle and varied. The SemEval-2023 Task 3 aimed to advance research in this area by focusing on three subtasks: news genre categorization, framing detection, and persuasion techniques detection across nine languages. This paper specifically investigates whether leveraging models trained on media bias detection can boost performance on the persuasion techniques detection task, hypothesizing that the two are closely related.
Methods
The core of the proposed approach is two-stage fine-tuning of multilingual transformer models. For Subtask 1 (News Genre Categorization), they used XLM-ROBERTa, and for Subtask 3 (Persuasion Techniques Detection), they used BERT-multilingual.
The "two-stage" aspect is key:
- Initial Fine-tuning on Media Bias Datasets: The models were first fine-tuned on the MBIC (Media Bias Annotation Dataset) and BABE (Bias in Articles Benchmark for English) datasets. These datasets are designed for media bias detection and contain annotated English articles. The idea here is to imbue the models with a general understanding of biased language and rhetorical devices.
- Task-Specific Fine-tuning: After the initial fine-tuning, the models were further fine-tuned on the specific SemEval-2023 Task 3 data for each subtask.
For Subtask 3, a cascading inference approach was employed. A first model predicted whether an article was biased or not. If predicted as biased, a second model then predicted the specific persuasion techniques used. This was an experimental choice, as they also explored using the biased/non-biased prediction as a feature for the second model, which yielded worse results.
Datasets
- MBIC (Media Bias Annotation Dataset): Used for initial fine-tuning of models for media bias detection.
- BABE (Bias in Articles Benchmark for English): Also used for initial fine-tuning, focusing on media bias detection.
- SemEval-2023 Task 3 Datasets: Provided for training, development, and testing across nine languages (English, French, German, Italian, Polish, Russian, Spanish, Greek, and Georgian). These datasets cover news genre, framing, and persuasion techniques.
Key Insights and Ideas
- Transfer Learning is Beneficial for Genre, Less So for Persuasion: The most significant finding is that transferring knowledge from media bias detection proved beneficial for news genre categorization, improving upon previous results. However, for the more nuanced task of detecting persuasive techniques, the gains were more modest.
- Model Struggles with Rare Classes: A consistent challenge across both tasks, but particularly for persuasion techniques, is the difficulty in detecting classes with fewer examples in the dataset. This suggests that the models are not robust enough to learn subtle patterns for less frequent phenomena.
- Cascading Inference for Persuasion Techniques: The authors found that a cascading approach (first bias detection, then persuasion technique detection) worked better than using the bias prediction as a feature for the persuasion model. This implies a degree of independence or a specific interaction that the cascading method captures better.
- Satire vs. Opinion Confusion: For genre detection, the system frequently confused satirical texts with opinion news, likely due to the overlap in opinionated claims within satirical content.
Key Results
- Subtask 1 (News Genre Categorization): Achieved a macro F1-score of 0.523 on the English test set, ranking 8th out of 22 teams.
- Subtask 3 (Persuasion Techniques Detection): Achieved a micro F1-score of 0.24 on the English test set, ranking 18th out of 23 teams.
- The paper also lists detailed F1-scores for individual persuasion techniques, showing high scores for common techniques like "Loaded Language" (0.65) and "Name Calling/Labeling" (0.63), but very low scores (0.00) for techniques like "Appeal to Popularity," "Appeal to Time," "Red Herring," and "Whataboutism."
Limitations and Future Work
- Limited Performance on Rare Techniques: The authors explicitly acknowledge that their system struggles with persuasion techniques that are less represented in the dataset. This is a major limitation for achieving high overall performance.
- Time Constraints: Due to time limitations, they were unable to explore a wider range of model architectures or fine-tuning strategies.
- Future Work: The authors plan to explore different architectures and more advanced transfer learning techniques. They also intend to leverage the media bias detection data more extensively, given its proven benefit for related tasks. Further investigation into handling class imbalance and exploring more sophisticated methods for capturing subtle linguistic differences is also suggested.
Summary updated