Appeal for Attention at SemEval-2023 Task 3: Data augmentation extension strategies for detection of online news persuasion techniques

Sergiu Amihaesei, Laura Cornei, George StoicaACL 2023

TL;DR

This paper tackles the challenge of detecting persuasion techniques in online news across multiple languages. The authors demonstrate that strategic data augmentation and extension techniques, particularly synonym replacement and translation-based methods, significantly improve the performance of a multilingual transformer model, leading to competitive results in a SemEval competition.

What Problem Does This Paper Address?

Identifying persuasion techniques in online news is crucial for combating manipulation, misinformation, and attacks on reputation. However, existing datasets for this task are often limited in size and language coverage, posing a significant challenge for developing robust multilingual models. Specifically, SemEval-2023 Task 3 focused on detecting these techniques across nine languages, with some languages only having test data available, pushing for zero-shot learning capabilities. The authors aim to address the data scarcity and imbalance issues that hinder model performance in such multilingual settings.

Methods

The core of this work lies in exploring various data augmentation and extension strategies to enhance a pre-trained multilingual transformer model, XLM-ROBERTa-Large.

  • Data Augmentation:
    • Synonym Replacement: A modified approach that considers part-of-speech tags and Levenshtein distance to replace words with synonyms from WordNet, aiming to create diverse yet semantically similar sentences.
  • Translation-Based Augmentation: Articles are translated from their original language into all other target languages using the DeepTranslator API. This helps balance language representation and increase data volume.
  • Data Extension:
    • Previous Competition Dataset: Incorporating data from SemEval-2021 Task 6, which focused on persuasion techniques in texts and memes (monolingual English).
  • Synthetic Data Generation: Utilizing GPT-3 (Text Davinci model) to generate new paragraphs containing specific persuasion techniques, with careful filtering to ensure quality and relevance.

The authors experimented with combinations of these techniques, fine-tuning the XLM-ROBERTa-Large model on the augmented and/or extended datasets.

Datasets

  • SemEval-2023 Task 3 Dataset: The primary dataset used for training and evaluation, covering nine languages: English, French, Italian, Spanish, German, Georgian, Greek, Polish, and Russian. Notably, only test data was available for Spanish, Georgian, and Greek, necessitating zero-shot learning approaches.
  • SemEval-2021 Task 6 Dataset: Used for data extension, specifically the textual data related to persuasion techniques. This dataset was monolingual (English).
  • GPT-3 Generated Data: Synthetic data created by the authors using GPT-3 to augment the training set.

Key Insights and Ideas

  • Data Quality and Quantity are Paramount: The study strongly suggests that the performance of even powerful models like XLM-ROBERTa-Large is heavily influenced by the quality and quantity of the training data.
  • Multilingual Augmentation is Effective: Translating existing data to other languages is a viable strategy for improving performance in low-resource languages.
  • Synonym Replacement Needs Nuance: A carefully implemented synonym replacement strategy, considering linguistic features, can generate useful augmented data.
  • GPT-3 as a Data Generator: GPT-3 can be a valuable tool for creating synthetic data, but careful filtering is essential to avoid generating definitions or irrelevant content.
  • Language-Specific Optimization: The optimal combination of augmentation and extension techniques can vary significantly across languages, highlighting the need for language-aware strategies.

Key Results

During the SemEval-2023 competition, using augmented data alone, the authors achieved:

  • 3rd place for English.
  • 13th place for Italian.
  • Ranks between 5th and 9th for the other seven languages.

Post-competition experiments exploring various combinations of augmentation and extension techniques revealed further improvements. For instance, the highest increase in F1 score was observed for Georgian, Italian, and German, with specific combinations of techniques yielding the best results for each language. The paper provides detailed F1 Micro scores for various combinations across languages in Table 3 and Appendix B.

Limitations and Future Work

  • English-Centric Resources: The authors note that the WordNet corpus for English is more developed than for other languages, potentially contributing to better performance with synonym-based augmentation in English.
  • Hyperparameter Tuning: Due to computational constraints, hyperparameter tuning was manual.
  • Future Work: The paper implicitly suggests further exploration into language-specific augmentation strategies and the impact of different GPT-3 prompting techniques for data generation. The authors also highlight the potential for further improvements by combining more advanced augmentation and extension methods.

Summary updated

BibTeX
@inproceedings{amihaesei2023appeal,
  author = {Sergiu Amihaesei and Laura Cornei and George Stoica},
  title = {{Appeal for Attention at SemEval-2023 Task 3: Data augmentation extension strategies for detection of online news persuasion techniques}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure