UMUTeam and SINAI at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation

José Antonio García-Díaz, Ronghao Pan, Salud María Jiménez Zafra, María-Teresa Martn-Valdivia, L. Alfonso Ureña-López, Rafael Valencia-GarcíaACL 2023

TL;DR

This paper details UMUTeam and SINAI's approach to the SemEval-2023 Task 9, which aims to predict the intimacy level of tweets across 10 languages. They employed an ensemble of three multilingual Large Language Models (LLMs) combined with data augmentation, achieving a respectable 30th place out of 45 teams, with notable success on unseen languages like Korean and Hindi.

What Problem Does This Paper Address?

The core challenge tackled is understanding and quantifying "intimacy" in short text, specifically tweets, across a diverse range of languages. Intimacy, defined as how individuals share personal data and emotions, is a nuanced concept that can be difficult to capture, especially in the brevity and informality of social media. The SemEval-2023 Task 9 (MTIA) specifically aimed to push the boundaries of this analysis by requiring participants to work with 10 languages, six of which were not included in the provided training data. This multilingual aspect, coupled with the constraint of submitting only a single run, presented a significant hurdle for developing robust and generalizable models.

Methods

The UMUTeam and SINAI approach centered on an ensemble learning strategy. They leveraged three powerful multilingual Large Language Models (LLMs): multilingual BERT, XLM, and mDeBERTa. To bolster the training data, they employed data augmentation by translating a large English dataset of Reddit questions (annotated with intimacy scores) into the six languages present in the MTIA training set, effectively adding over 21,000 new sentences. This augmented dataset, along with the original MTIA training data, was used to train the LLMs. The final prediction was an average of the outputs from these three LLMs. While they also explored linguistic features using UMUTextStats, these were ultimately excluded from the final submission due to limitations in language coverage.

Datasets

The primary dataset used was the MINT dataset provided for SemEval-2023 Task 9, which contains 9491 tweets rated on an intimacy scale from 1 to 5. This dataset covered six languages for training: English, Spanish, Italian, Portuguese, French, and Chinese. For evaluation, the task included four additional languages: Hindi, Arabic, Dutch, and Korean. Additionally, the authors incorporated a dataset from Pei and Jurgens (2020), consisting of over 2,000 English questions from Reddit, which was translated into the six training languages as part of their data augmentation strategy.

Key Insights and Ideas

  • Ensemble Power: Combining predictions from multiple LLMs proved more effective than any single model, demonstrating the benefit of diverse model perspectives.
  • Data Augmentation's Role: Translating external datasets significantly boosted performance, especially for languages with limited training data, highlighting the value of creative data expansion.
  • Unseen Languages as a Challenge: The model performed surprisingly well on unseen languages like Korean and Hindi, suggesting that the multilingual LLMs learned robust cross-lingual representations, though performance varied across these languages.
  • Intimacy Distribution Skew: The majority of tweets in the dataset received low intimacy scores, indicating a potential bias in the data and a challenge for models to accurately predict higher intimacy levels.

Key Results

The UMUTeam and SINAI achieved a Pearson's R score of 0.532 on the official leaderboard, ranking 30th out of 45 participating teams. Their best performance was on the unseen languages, with Korean at 16th place and Hindi at 19th. While their custom validation split yielded a higher Pearson's R of 0.682, the official score reflects the challenge of generalizing to unseen data. The ablation study indicated that data augmentation generally improved performance, though the impact was modest for some metrics like RMSE.

Limitations and Future Work

The authors acknowledge several limitations. Firstly, their fluency in all 10 languages was limited, potentially leading to missed nuances in context and cultural understanding. Secondly, they question whether the data augmentation process was truly beneficial, as translations might not perfectly capture cultural context. Future work could involve a more in-depth dataset analysis, potentially translating all sentences into a common language and incorporating topic-related features. They also plan to implement nested cross-validation to mitigate bias from their custom validation split and explore other modules like preprocessing.

Summary updated

BibTeX
@inproceedings{garciadiaz2023umuteam,
  author = {José Antonio García-Díaz and Ronghao Pan and Salud María Jiménez Zafra and María-Teresa Martn-Valdivia and L. Alfonso Ureña-López and Rafael Valencia-García},
  title = {{UMUTeam and SINAI at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure