KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection

Timo Hromadka, Timotej Smolen, Tomas Remis, Branislav Pecher, Ivan SrbaACL 2023

TL;DR

This paper presents a winning system for detecting persuasion techniques in multilingual text, achieving top ranks across several languages. The team found that a single, large multilingual model (XLM-ROBERTa large) fine-tuned on all available data, with carefully calibrated confidence thresholds, outperformed other strategies, including monolingual models and ensembles, especially for unseen languages.

What Problem Does This Paper Address?

The SemEval-2023 Task 3 focuses on identifying persuasion techniques within text, a challenging multi-label classification problem with 23 possible techniques. A significant hurdle is the multilingual nature of the dataset, spanning six languages (English, French, German, Italian, Polish, Russian) and including three "surprise" languages (Greek, Georgian, Spanish) for which no training data was provided. This multilingualism, coupled with a large number of labels, leads to data sparsity for certain language-label combinations, making it difficult to train effective models, especially individual monolingual ones. The task's importance lies in its potential to improve content credibility assessment and disinformation detection.

Methods

The core of the KInITVeraAI system is fine-tuning a large pre-trained transformer-based language model. The authors explored several strategies:

  • Candidate Model Selection: They compared monolingual models (BERT, RoBERTa) trained on translated English data against multilingual models (mBERT, XLM-ROBERTa).
  • Multilinguality Strategies: They evaluated monolingual models, multilingual models, and their ensembles.
  • Confidence Threshold Calibration: A crucial step involved finding the optimal confidence threshold for predicting labels. This was done separately for "seen" (trained on) and "surprise" (unseen) languages, simulating a zero-shot setting.
  • Preprocessing Strategies: They investigated the impact of text cleaning and normalization.
  • Layer Freezing: They experimented with freezing different portions of the pre-trained model layers during fine-tuning.

The final system opted for a single, large multilingual model (XLM-ROBERTa large) fine-tuned on all languages simultaneously, with no preprocessing and no layer freezing, but with carefully calibrated confidence thresholds.

Datasets

The primary dataset used was the official SemEval-2023 Task 3 dataset. This dataset consists of paragraphs from news articles in 6 languages (English, French, German, Italian, Polish, Russian) collected between 2020 and mid-2022. The task also included 3 surprise languages (Greek, Georgian, Spanish) for which no labeled training data was available. The authors note a significant data imbalance across both classes (persuasion techniques) and languages.

Key Insights and Ideas

  • Multilingual Models Trump Monolingual (with caveats): While monolingual models trained on translated data can perform well, a single, large multilingual model (XLM-ROBERTa large) fine-tuned on all languages simultaneously proved superior, especially for the challenging zero-shot (unseen language) scenarios.
  • Confidence Thresholds are Critical: Calibrating confidence thresholds is not just a minor tuning step; it significantly impacts performance, particularly for unseen languages where models are less certain. The optimal threshold differed between seen and unseen languages.
  • Simplicity Can Be Powerful: Despite exploring various complex strategies like preprocessing and layer freezing, the most effective solution was a relatively simple fine-tuning approach with a well-chosen model and calibrated thresholds.
  • Data Imbalance is a Major Challenge: The inherent data imbalance across languages and persuasion techniques significantly limits the effectiveness of training multiple monolingual models and complicates evaluation, especially with the F1 micro metric.
  • Preprocessing and Layer Freezing Offer Little Gain: Surprisingly, common techniques like text preprocessing and freezing most layers of the pre-trained model did not improve performance and, in some cases (layer freezing), even degraded it.

Key Results

The KInITVeraAI system achieved outstanding results:

  • Top Ranks: The system ranked 1st for 6 out of 9 languages (Italian, Russian, German, Polish, Greek, and Georgian).
  • Surprise Language Performance: It performed exceptionally well on the surprise languages, achieving 1st place for Greek and Georgian and 2nd place for Spanish.
  • Overall Performance: The best performing model, XLM-ROBERTa large, achieved an F1 micro score of 45.09% in the multilingual setting. The ensemble of RoBERTa large and XLM-ROBERTa large achieved 47.66%.
  • Confidence Threshold Impact: Calibrating the confidence threshold to 0.29 for seen languages and 0.28 for unseen languages (in the final submission) was crucial for achieving these results.

Limitations and Future Work

  • Data Imbalance: The authors acknowledge that the data imbalance remains a significant challenge, leading to the model focusing more on majority classes and struggling with rarer persuasion techniques.
  • Zero-Shot Performance: While strong, the performance on unseen languages is still lower than on seen languages, indicating room for improvement.
  • Future Directions: The team plans to investigate the potential of prompting and in-context learning with larger language models (like GPT-3 or ChatGPT) to potentially achieve deeper text understanding. They also aim to address potential biases towards majority classes and the instability issues associated with these advanced approaches.

Summary updated

BibTeX
@inproceedings{hromadka2023kinitveraai,
  author = {Timo Hromadka and Timotej Smolen and Tomas Remis and Branislav Pecher and Ivan Srba},
  title = {{KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure