TL;DR
This paper presents DSHacker's approach to SemEval-2023 Task 3, focusing on detecting news genres and persuasion techniques across multiple languages. By leveraging data augmentation techniques like machine translation (MT) and text generation (GPT-3), they achieved top-10 rankings across all languages, with particularly strong performance in genre detection.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically identifying two key aspects of news articles: their genre (satire, opinion, or reporting) and the presence of persuasion techniques. This is crucial for understanding information quality, categorizing content, and combating misinformation. The task is made more complex by its multilingual nature, requiring systems that can perform these tasks across various languages, often with limited labeled data for each.
Methods
The DSHacker team employed a multi-pronged approach, with distinct strategies for genre detection and persuasion technique detection.
For genre detection, they utilized a single, powerful multilingual model: XLM-RoBERTa-large. To boost its performance, they augmented the training data by generating synthetic articles for each genre using OpenAI's GPT-3 Davinci model. This allowed the model to learn from a more diverse set of examples.
For persuasion techniques detection, their approach varied by language. For Polish, they combined classical machine learning models (XGBoost, Logistic Regression, LightGBM) with advanced features derived from HerBERT embeddings and linguistic analysis using the StyloMetrix library. For other languages (English, Italian, Russian, French, German), they relied on transformer-based models. A key augmentation technique here was machine translation using DeepL, translating existing paragraphs to create more training data. For longer paragraphs, they also employed a summarization pipeline using the t5-small model.
A significant post-deadline effort involved developing a multilingual persuasion detection system using XLM-RoBERTa-large, which outperformed their initial language-specific models.
Datasets
The study used datasets provided for SemEval-2023 Task 3, which included news articles in English, French, German, Italian, Polish, and Russian for training and development. The test set also included articles in Spanish, Greek, and Georgian.
The authors also generated new data through:
- GPT-3 Davinci: For genre detection, synthetic articles were generated for each genre.
- DeepL Machine Translation: For persuasion techniques detection, existing paragraphs were translated between languages to augment the training data.
Key Insights and Ideas
- Data Augmentation is Key: Both machine translation (DeepL) and text generation (GPT-3) proved highly effective in improving model performance for both genre and persuasion technique detection, especially in multilingual settings.
- Multilingual Models Shine for Genre: A single, large multilingual model (XLM-RoBERTa-large) was sufficient and highly effective for genre detection across all languages, highlighting the power of these models in capturing cross-lingual linguistic patterns.
- Language-Specific Nuances for Persuasion: While a multilingual approach eventually proved superior for persuasion detection, initial experiments showed that language-specific models and feature engineering (especially for Polish) could yield competitive results.
- Post-Deadline Improvements: The post-deadline multilingual XLM-RoBERTa-large model for persuasion detection significantly outperformed the initial language-specific submissions, suggesting that larger, more general models can be very effective even for tasks with language-specific characteristics.
Key Results
- Genre Detection: DSHacker achieved excellent results, securing 1st place in Spanish, 2nd in German, and 3rd in Italian for genre detection. Their models consistently ranked among the top performers.
- Persuasion Techniques Detection: They achieved the highest F1 micro score for Italian and secured the 3rd official rank in Polish. Their systems consistently ranked in the top 10 across languages. The post-deadline multilingual model also showed superior performance.
Limitations and Future Work
- Strict SemEval Format: The authors faced limitations due to the strict SemEval format, which constrained the scope and depth of their analysis, forcing them to prioritize certain methods.
- Limited Augmentation Combinations: Due to time constraints, they couldn't explore all possible combinations of data augmentation techniques.
- Translation Scope: For persuasion techniques, they only translated certain languages into others for training and used StyloMetrix features exclusively for Polish.
- OpenAI Policy: Generating synthetic data for persuasion techniques was restricted by OpenAI's policy due to potential misuse.
- Future Work: The authors suggest further developing the multilingual approach for persuasion techniques detection and a more thorough investigation into the impact of various augmentation techniques on model performance.
Summary updated