TL;DR
This paper tackles the challenge of detecting propaganda and persuasion techniques in online news, particularly in low-data scenarios. They introduce a novel approach using Shapley Additive Explanations (SHAP), an explainable AI (XAI) technique, to improve model performance by identifying crucial features and recalibrating attention mechanisms. Their method achieved competitive results in the SemEval-2023 Task 3, highlighting the potential of XAI for enhancing NLP tasks.
What Problem Does This Paper Address?
The proliferation of online news has unfortunately led to an increase in sophisticated propaganda and persuasive content. Identifying these techniques is crucial for media literacy and combating misinformation. However, accurately detecting propaganda and persuasion methods is a significant challenge, especially when dealing with limited labeled training data. Existing methods often struggle to generalize well in such scenarios. SemEval-2023 Task 3 specifically aimed to foster the development of tools to analyze online media content, focusing on genre classification (opinion, report, satire) and the identification of 23 distinct persuasion techniques. This paper addresses the limitations of current approaches by proposing a method that leverages explainability to overcome data scarcity and improve model robustness.
Methods
The core of the proposed solution lies in the application of Shapley Additive Explanations (SHAP), an XAI technique that quantifies the contribution of each feature to a model's prediction. The authors employ SHAP in two distinct ways for the two subtasks:
- Subtask 1: News Genre Categorisation: SHAP is used for feature selection. By understanding which tokens drive the model's classification decisions, they can identify and filter out tokens that lead to incorrect predictions. This helps in refining the input data and improving the model's focus.
- Subtask 3: Persuasion Techniques Detection: Here, SHAP is used to re-calibrate the Attention Mechanism within the Transformer model. The idea is to identify words that are highly characteristic of specific persuasion techniques. These "SHAP Vocabularies" are then used to guide the model's attention, helping it to focus on the most relevant parts of the text for accurate persuasion detection.
The underlying model architecture for both subtasks is the DistilBERT transformer, a smaller, faster, and more efficient version of BERT. For subtask 3, they employ a two-stage approach: first, a binary classifier to detect if a paragraph contains any persuasion, and then a multiclass classifier to identify the specific techniques.
Datasets
The experiments were conducted using the dataset provided by the SemEval-2023 Task 3 organizers for the English subtasks.
- Subtask 1 (News Genre Categorisation): The English training dataset contained 433 news and web articles. To address class imbalance (particularly for satire), data augmentation was performed by translating articles from other languages (Italian, French, German, Polish, Russian) into English. The final training set comprised 674 articles.
- Subtask 3 (Persuasion Techniques Detection): The English dataset for this subtask included 446 training news articles and 9498 paragraphs. A significant portion (5738 paragraphs) were labeled as "No Persuasion," while 3760 fell under at least one of the 23 persuasion techniques.
Key Insights and Ideas
- XAI for Data Scarcity: The paper strongly advocates for using XAI techniques like SHAP not just for understanding model behavior but as a practical tool to overcome limitations imposed by scarce labeled data.
- Targeted Feature Selection: SHAP's ability to pinpoint influential features allows for a more intelligent preprocessing step, effectively filtering out noise and focusing the model on relevant information.
- Attention Mechanism Refinement: By identifying key words associated with persuasion techniques, SHAP can guide the attention mechanism to focus on these critical elements, leading to more accurate classification.
- Two-Stage Classification for Persuasion: The strategy of using a binary "persuasion vs. no persuasion" classifier before a multiclass classifier for specific techniques is a pragmatic approach to handle the imbalance and complexity of the persuasion detection task.
Key Results
- Subtask 1 (News Genre Categorisation): The BERT-SHAP approach achieved a macro F1-score of 58.621%, securing 6th place out of 23 teams on the English leaderboard. This demonstrates competitive performance, especially considering the challenges of class imbalance.
- Subtask 3 (Persuasion Techniques Detection): The BERT-SHAP method achieved a micro F1-score of 29.758%, ranking 12th out of 23 teams on the English leaderboard. While this score is lower than the top performers, the paper highlights the inherent difficulty of the task, particularly with many techniques exhibiting very low performance (some at 0%) due to data imbalance.
Limitations and Future Work
- Low Performance on Minor Classes/Techniques: The authors acknowledge that even with their approach, minor classes in genre categorization and many persuasion techniques in subtask 3 still exhibit poor performance. This is attributed to the inherent data imbalance.
- SHAP Vocabularies Construction: For persuasion detection, the authors suggest that future work could improve the construction of SHAP Vocabularies by incorporating spans of text that contain the actual persuasion attempt, rather than just individual words. This could allow for a more focused analysis by the binary classifiers.
- Addressing Specific Techniques: For techniques like "Repetition," which involve repeated word usage, the authors suggest that defining specialized models or rules might be more effective than relying solely on the general multiclass classifier.
- Further Investigation: The paper concludes that the experimental results are promising and encourage further investigation into XAI-based approaches for persuasion detection.
Summary updated