NAP at SemEval-2023 Task 3: Is Less Really More? (Back-)Translation as Data Augmentation Strategies for Detecting Persuasion Techniques

Neele Falk, Annerose Eichel, Prisca PiccirilliACL 2023

TL;DR

This paper tackles the challenge of detecting persuasion techniques in multilingual news articles, a task hindered by limited training data. The authors demonstrate that data augmentation through back-translation significantly boosts performance across multiple languages, but caution that excessive augmentation can be detrimental.

What Problem Does This Paper Address?

Detecting persuasion techniques in text is crucial for understanding how information is framed and how audiences are influenced. However, this task becomes significantly more challenging in a multilingual setting due to the scarcity of labeled data across different languages. SemEval-2023 Task 3 specifically aimed to identify persuasion techniques in news articles across nine languages, including three "surprise" languages with only test data. The core problem is how to effectively leverage limited data to build robust, language-agnostic models for this task.

Methods

The authors employed a system that combines multiple state-of-the-art multilingual transformer models in an ensemble. Their primary contribution lies in exploring data augmentation strategies, specifically focusing on (back-)translation.

  • Data Augmentation: They generated translations and back-translations between all available language pairs to increase the training data size. Back-translation involves translating text to another language and then back to the original language, aiming to create paraphrased versions that retain the original meaning and persuasive intent.
  • Multilingual Transformer Models: They utilized XLM-RoBERTa (XLM-R) models, fine-tuning them with a multi-label classification head.
  • Adapters: For specific persuasion techniques, they trained label-specific adapters, which are small, efficient parameter sets inserted into transformer layers. This approach is particularly useful for low-frequency labels.
  • SetFit: A few-shot learning method based on sentence transformers was also employed.
  • Ensembling: Predictions from various models were combined using an ensemble approach, with threshold moving and rule-based heuristics applied for optimization.

Datasets

The primary dataset used was the SemEval-2023 Task 3 gold dataset, which contains news articles annotated with persuasion techniques across six languages (English, German, French, Italian, Polish, Russian) and three surprise languages (Spanish, Greek, Georgian). The authors augmented this dataset by generating translations and back-translations. They also utilized textual spans associated with persuasion techniques for some experiments.

Key Insights and Ideas

  • (Back-)Translation is a Powerful Augmentation Tool: The study strongly suggests that (back-)translation is a highly effective strategy for increasing training data and improving performance in multilingual persuasion technique detection.
  • "Less is More" Principle: While more data generally helps, the paper highlights that simply adding a large volume of machine-generated translations can actually hurt performance. A careful balance between original and augmented data is crucial.
  • Back-Translation Outperforms Direct Translation: Surprisingly, back-translation consistently improved performance across most labels, while direct translation often hindered it, even when combined with back-translation. This suggests that the paraphrasing aspect of back-translation is more beneficial than simple translation.
  • Language and Culture Dependence: Human evaluation revealed that some persuasion techniques are more language- and culture-dependent than others, impacting their transferability and preservation during translation.
  • Adapters Excel for Rare Labels: The adapter-based approach proved particularly effective for detecting less frequent persuasion techniques, demonstrating the utility of specialized modules.

Key Results

The system achieved significant success in the SemEval-2023 Task 3, winning the task for French and securing second place for German, Italian, and Polish. They also achieved third place for the surprise languages Spanish, Greek, and Georgian.

  • Performance Boost: Data augmentation via (back-)translation led to substantial performance improvements, especially for under-represented labels.
  • Human Evaluation Insights: Human evaluation indicated that while fluency was generally maintained, direct translations sometimes resulted in a lower percentage of "human-produced" text. Fidelity and surface variability scores suggested that paraphrases from back-translation were reasonably faithful and varied.
  • Regression Analysis: A regression model showed that the choice of augmentation strategy significantly impacted performance, with the interaction between training data and labels explaining most of the variance.

Limitations and Future Work

  • Human Evaluation Scale: The authors acknowledge that their small-scale human annotation study might limit the generalizability of findings regarding the quality of (back-)translations. A larger-scale study is suggested for more robust conclusions.
  • Surprise Language Data: The surprise languages only had test data, limiting the extent to which augmentation strategies could be fully explored and validated for them during training.
  • Investigating Translation vs. Back-Translation Discrepancy: The paper notes the surprising difference in performance between direct translation and back-translation and plans further analysis to understand this phenomenon.
  • Language Family Grouping: The paper hints at potential improvements by training on languages grouped by their families, suggesting this as a promising direction for future research.

Summary updated

BibTeX
@inproceedings{falk2023nap,
  author = {Neele Falk and Annerose Eichel and Prisca Piccirilli},
  title = {{NAP at SemEval-2023 Task 3: Is Less Really More? (Back-)Translation as Data Augmentation Strategies for Detecting Persuasion Techniques}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure