CLaC at SemEval-2023 Task 3: Language Potluck RoBERTa Detects Online Persuasion Techniques in a Multilingual Setup

Nelson Filipe Costa, Bryce Hamilton, Leila KosseimACL 2023

TL;DR

This paper tackles the challenge of automatically detecting online persuasion techniques across multiple languages. The authors adapted a RoBERTa-base model, primarily trained on English, for this multilingual task, finding that while it excelled in some languages, a purely English-centric approach wasn't universally effective. This highlights the complexities of cross-lingual generalization in natural language processing.

What Problem Does This Paper Address?

The proliferation of "fake news" and misinformation, often employing sophisticated rhetorical and psychological persuasion techniques, poses a significant societal challenge. Automatically identifying these techniques is crucial for understanding and combating their influence. SemEval-2023 Task 3 specifically focuses on detecting these persuasion techniques in online news articles across various languages. The challenge lies in building a robust system that can generalize across linguistic boundaries, especially when training data is limited for many languages.

Methods

The core of the authors' approach is a RoBERTa-base model, a powerful Transformer-based language model. They framed the problem as a 23-class multi-label binary classification task, where each paragraph of a news article could be labeled with one or more of the 23 identified persuasion techniques.

A key methodological decision was their multilingual strategy: instead of training separate models for each language, they translated all non-English training data into English and then trained a single RoBERTa-base model. This was motivated by initial experiments suggesting potential for good cross-lingual generalization.

To address data imbalances and missing techniques within languages, they employed data augmentation. This involved translating instances of under-represented techniques from other languages into the target language. They also implemented strategies to drop over-represented instances to create a more balanced dataset.

Datasets

The study utilized datasets provided for SemEval-2023 Task 3. These datasets contained news articles annotated at the paragraph level for 23 different persuasion techniques. The training data was initially available for six languages: English, French, German, Italian, Polish, and Russian. Three additional "surprise" languages for testing were included: Georgian, Greek, and Spanish.

Key Insights and Ideas

  • The Limits of English-Centric Generalization: The most striking insight is that a RoBERTa model trained predominantly on English, even with data augmentation and translation, did not consistently generalize well across all languages. While it performed well in English, Georgian, and Greek, its performance suffered in other languages. This suggests that linguistic nuances and structural differences between languages are significant hurdles for purely translation-based cross-lingual transfer.
  • Data Augmentation's Double-Edged Sword: While data augmentation helped to balance technique distributions and introduce missing categories, it also introduced complexities. The authors had to carefully manage over-represented techniques that co-occurred with augmented instances, indicating a delicate balance in this process.
  • The Importance of Native Language Training: The results strongly suggest that training models on data native to the target language (or at least languages with closer linguistic ties) is likely more effective than relying solely on translation to a dominant language like English.

Key Results

  • The RoBERTa-base model, trained on translated English data, significantly outperformed the baseline in English, Georgian, and Greek.
  • For the other six languages (French, German, Italian, Polish, Russian), the model either performed worse or only slightly better than the baseline.
  • In a separate experiment (Table 2), training a RoBERTa-base model on English and testing it on other languages showed particularly good performance on French and Italian, hinting at potential etymological connections influencing generalization.

Limitations and Future Work

The authors acknowledge that their assumption about the strong generalizability of an English-trained model was proven wrong by the official leaderboard results. They note that the performance drop in several languages could be due to differences in label distributions between training and test sets, or inherent linguistic differences.

Future work proposed includes:

  • Exploring other language models like ALBERT.
  • Evaluating the performance of models trained on the native language of the prediction task.
  • Investigating a cascading model of classification systems for each persuasion technique (one-vs-all).
  • Analyzing which techniques are more frequently misclassified across languages to understand how subjective information might be lost through machine translation.

Summary updated

BibTeX
@inproceedings{costa2023clac,
  author = {Nelson Filipe Costa and Bryce Hamilton and Leila Kosseim},
  title = {{CLaC at SemEval-2023 Task 3: Language Potluck RoBERTa Detects Online Persuasion Techniques in a Multilingual Setup}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure