ReDASPersuasion at SemEval-2023 Task 3: Persuasion Detection using Multilingual Transformers and Language Agnostic Features

Fatima Zahra Qachfar, Rakesh VermaACL 2023

TL;DR

This paper presents a multilingual system for detecting persuasion techniques in text, outperforming the SemEval baseline across most languages. It achieves this by combining a powerful multilingual transformer (XLM-ROBERTa) with novel language-agnostic features derived from sentiment analysis, exaggeration detection, and profanity filtering, making it robust across 100 languages.

What Problem Does This Paper Address?

Persuasion is a pervasive element in communication, yet automatically detecting its various techniques, especially across multiple languages, remains a significant challenge. SemEval-2023 Task 3 specifically aimed to address this by tasking participants with identifying one or more of 23 persuasion techniques in text data from six initial languages, with three surprise languages added for zero-shot evaluation. The core difficulty lies in building a system that can generalize across diverse linguistic structures and cultural nuances, while also handling the inherent imbalanced nature of persuasion technique occurrences in real-world text.

Methods

The ReDASPersuasion system employs a three-module architecture:

  1. Multilingual Transformer Model: The backbone of the system is XLM-ROBERTa, a pre-trained multilingual transformer model known for its strong performance on cross-lingual tasks. This module handles the primary text classification.
  2. Language-Agnostic Feature Engineering: This is a key innovation. The authors engineer several features designed to be independent of specific languages. These include:
  3. Appeal to Fear: Detected using sentiment analysis (polarity) via the Polyglot library, which supports over 136 languages.
  4. Exaggeration and Minimization: Identified by analyzing POS tags, adverbs, indefinite pronouns, and quantifiers. They also leverage the HyperProbe dataset and TextBlob for subjectivity and polarity extraction.
  5. Loaded Language: Detected using a profanity-filter package, extended to support Russian and other languages by incorporating Hunspell dictionaries.
  6. Time: A new feature introduced to capture temporal adverbs, using the dateparser Python package.
  7. Redundancy and Specificity: Extracted using the MoreThanSentiments package to identify repeated n-grams and specific entity counts.
  8. Multi-label Classification Head: The [CLS] embeddings from the transformer are concatenated with the engineered persuasion features. This combined representation is then fed through a dropout layer and a dense linear layer with sigmoid activation outputs to predict the presence of each of the 23 persuasion techniques independently.

Data augmentation techniques, both static (synonym replacement via PPDB) and dynamic (contextualized word embeddings via nlpaug), were also employed to increase the training data size.

Datasets

The primary dataset used is from SemEval-2023 Task 3, specifically for Subtask 3 (Persuasion Techniques Detection). This dataset includes text samples in English, French, Polish, Italian, Russian, and German, with additional zero-shot evaluation for Spanish, Greek, and Georgian. The authors also mention using the HyperProbe dataset for identifying exaggerations and WordNet for synonym selection.

Key Insights and Ideas

  • Language-Agnostic Features are Crucial: The success of the system hinges on the carefully engineered language-agnostic features. These features capture semantic and stylistic cues that are transferable across languages, complementing the multilingual transformer's capabilities.
  • Sentiment Analysis as a Proxy for Persuasion: The finding that sentiment analysis (polarity and subjectivity) is a strong indicator for techniques like "Appeal to Fear," "Loaded Language," and "Slogans" highlights the interconnectedness of emotional appeals and persuasion.
  • Data Augmentation Boosts Performance: Both static and dynamic data augmentation significantly improved the system's performance, particularly for languages like Polish, demonstrating the benefit of increasing training data diversity.
  • Model Size Matters: The transition from XLM-RoBERTa-base to XLM-RoBERTa-large led to performance improvements, suggesting that larger, more capable models can better leverage the complex features.
  • Imbalanced Data is a Persistent Challenge: The paper acknowledges the imbalanced nature of persuasion technique occurrences, which negatively impacts detection for less frequent techniques.

Key Results

The ReDASPersuasion system achieved strong results on the SemEval-2023 Task 3.

  • It outperformed the SemEval baseline in all languages except the zero-shot prediction languages.
  • The system achieved the highest F1-Micro score of 0.45 for Italian, securing the eighth position on the leaderboard.
  • For Polish, a 66.4% increase in performance was observed after post-deadline improvements and data augmentation compared to the pre-deadline model.
  • The XLM-RoBERTa-large model, combined with fine-tuning and data augmentation, significantly enhanced classification across all languages.

Limitations and Future Work

  • Limited Zero-Shot Performance: The system's performance on the surprise zero-shot languages (Spanish, Greek, Georgian) was not the primary focus and was not as strong as in the initial six languages.
  • Imbalanced Data: The imbalanced distribution of persuasion techniques in the dataset poses a challenge for detecting less frequent ones.
  • Specific Technique Failures: The system struggled to identify techniques like "obfuscation," "straw man," and "whataboutism," which were present in only a few examples in the training and development sets.

Future work suggested includes:

  • Exploring Zero-Shot Learning: Investigating how to better leverage persuasion attributes for zero-shot classification tasks using transfer learning.
  • Developing New Features: Focusing on qualitative data augmentation and creating new features to target the under-detected persuasion techniques.
  • Handling Longer Sequences: Exploring models like BigBird and Longformer to handle longer text sequences beyond the typical 512-token limit of XLM-R, which could improve performance on documents with extensive persuasive content.
  • Interpreting Persuasion: Further research into the interpretability of persuasive text per class to better understand the linguistic indicators of persuasion.

Summary updated

BibTeX
@inproceedings{qachfar2023redaspersuasion,
  author = {Fatima Zahra Qachfar and Rakesh Verma},
  title = {{ReDASPersuasion at SemEval-2023 Task 3: Persuasion Detection using Multilingual Transformers and Language Agnostic Features}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure