HULAT at SemEval-2023 Task 9: Data Augmentation for Pre-trained Transformers Applied to Multilingual Tweet Intimacy Analysis

Isabel Segura-BedmarACL 2023

TL;DR

This paper explores the effectiveness of data augmentation techniques for multilingual tweet intimacy analysis using pre-trained transformer models. While data augmentation showed only marginal improvements, the XLM-T model achieved a respectable 27th place out of 45 systems, demonstrating strong performance in Portuguese and English.

What Problem Does This Paper Address?

The paper tackles the novel task of automatically estimating the level of intimacy in multilingual tweets. Understanding textual intimacy is crucial for social sciences, marketing, and even political analysis. However, research in this area is limited, and the multilingual aspect adds significant complexity. The MINT dataset, used for this task, comprises tweets in 10 languages, with a particular challenge being the inclusion of "unseen" languages (Arabic, Dutch, Korean, Hindi) in the test set that were not present in the training set. This necessitates models that can generalize across languages and potentially adapt to new linguistic contexts.

Methods

The researchers fine-tuned several pre-trained transformer models, including BERT, DistilBERT, MiniLM, XLM-ROBERTa (XLM-R), and XLM-T, on the MINT dataset. They also investigated the impact of data augmentation techniques, specifically those provided by EDA (Synonym Replacement, Random Insertion, Random Swap, Random Deletion) and NLPAug (Character level, Word level, Sentence level augmentation). The primary goal was to identify the best model and augmentation strategy for predicting intimacy scores, which are treated as a regression task on a scale of 1 to 5.

Datasets

The primary dataset used is the MINT dataset (Pei and Jurgens, 2020), which contains 13,384 tweets in 10 languages. The training subset includes tweets in English, French, Spanish, Italian, Portuguese, and Chinese. The test dataset includes these languages plus Arabic, Dutch, Korean, and Hindi. The authors also mention using a portion of the training data for validation and testing during the development phase.

Key Insights and Ideas

  • XLM-T Shines: The XLM-T model, pre-trained on a large corpus of tweets in over thirty languages, emerged as the best-performing model, outperforming other multilingual transformers like XLM-R. This highlights the benefit of domain-specific pre-training for this task.
  • Data Augmentation's Modest Impact: While data augmentation was explored to increase the training data, its contribution to performance improvement was minimal. In some cases, it even slightly decreased performance for certain models. This suggests that for this specific task and dataset, synthetic data generation might not be as beneficial as initially hoped.
  • Language Matters: The performance varied significantly across languages, with the model performing much better on "seen" languages (those present in the training set) compared to "unseen" ones. This underscores the challenge of cross-lingual transfer learning for nuanced tasks like intimacy analysis.
  • Tweet Length and Language: The study observed correlations between tweet length and language, with ideographic languages like Chinese and Korean exhibiting shorter tweets, which is a known linguistic phenomenon.

Key Results

The HULAT team achieved an overall Pearson correlation of 0.55, ranking 27th out of 45 participating systems in SemEval-2023 Task 9. For Portuguese and English, their system performed particularly well, ranking third and ninth respectively. For the unseen languages, the overall correlation was lower (0.355), but the system ranked eighth for Dutch. The best performing model was XLM-T, achieving a Pearson score of 0.563 with data augmentation.

Limitations and Future Work

The authors acknowledge that their results for unseen languages are modest and that there is significant room for improvement, especially in Korean. Future work includes:

  • Zero-shot Learning: Exploring zero-shot text classification methods to automatically annotate unseen languages, followed by manual review.
  • Dataset Expansion: Augmenting the dataset with texts from novels (higher intimacy) and academic books (lower intimacy) to create a more diverse and comprehensive resource.
  • Advanced Data Augmentation: Investigating more sophisticated data augmentation techniques, particularly back-translation, to generate synthetic data for all languages.

Summary updated

BibTeX
@inproceedings{segurabedmar2023hulat,
  author = {Isabel Segura-Bedmar},
  title = {{HULAT at SemEval-2023 Task 9: Data Augmentation for Pre-trained Transformers Applied to Multilingual Tweet Intimacy Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure