ROZAM at SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis

Mohammadmostafa Rostamkhani, Ghazal Zamaninejad, Sauleh EetemadiACL 2023

TL;DR

This paper tackles the challenge of automatically detecting intimacy in multilingual tweets using the XLM-T language model. By fine-tuning XLM-T on the MINT dataset and experimenting with various configurations, they achieved a respectable Pearson Correlation of 0.52, demonstrating the feasibility of this task across different languages.

What Problem Does This Paper Address?

Understanding the nuances of human relationships is crucial, and intimacy is a key dimension. While intimacy plays a significant role in language, datasets for its analysis are scarce. Existing textual intimacy datasets are often limited to a single language, hindering the development of models that can generalize across diverse linguistic contexts. This paper addresses this gap by focusing on multilingual tweet intimacy analysis, aiming to predict intimacy scores for tweets in multiple languages. This capability could unlock insights into social dynamics and communication patterns across cultures.

Methods

The core of the ROZAM system is the XLM-T (Cross-lingual Language Model - Twitter) model, a multilingual pre-trained language model specifically trained on a large corpus of tweets. The authors fine-tuned XLM-T for a regression task, aiming to predict an intimacy score between 1 and 5. They experimented with several deep learning architectures, primarily by modifying the number of XLM-T layers that were frozen during fine-tuning, and also explored different loss functions (MSE, negative Pearson Correlation, and a combination of both) to optimize performance. Data augmentation techniques, including translating existing tweets to zero-shot languages, were employed to improve generalization.

Datasets

The primary dataset used is the MINT (Multilingual INTmacy) dataset. This dataset contains tweets annotated with intimacy scores ranging from 1 to 5. It covers 10 languages, with 6 languages used for training (English, Spanish, Portuguese, Italian, French, Chinese) and 4 languages for zero-shot evaluation (Hindi, Korean, Dutch, and Arabic). The authors also mention using an "original" dataset in their initial experiments, which likely refers to a subset or a different version of the MINT dataset.

Key Insights and Ideas

  • XLM-T is a strong foundation: The choice of XLM-T, pre-trained on a massive multilingual tweet corpus, proved to be a robust starting point for this task. Its inherent understanding of tweet language and multilingualism was key.
  • Fine-tuning strategy matters: Experimenting with freezing different layers of XLM-T revealed that fine-tuning the later layers, rather than the entire model or just the very last layers, yielded better results. This suggests a balance between leveraging pre-trained knowledge and adapting to the specific intimacy prediction task.
  • Loss function trade-offs: Combining MSE with Pearson Correlation as a loss function was found to be beneficial, aiming for both accurate predictions (MSE) and high correlation with the true intimacy scores (Pearson Correlation).
  • Data augmentation is crucial for zero-shot: Translating tweets to zero-shot languages was a vital strategy to improve the model's performance on languages it hadn't explicitly seen during training. This highlights the importance of creative data augmentation for cross-lingual tasks.

Key Results

The ROZAM system achieved a Pearson Correlation score of 0.52 on the overall task, ranking 33rd out of 45 teams at SemEval-2023 Task 9. Their best performing model involved fine-tuning the last few layers of XLM-T with a combination of MSE and Pearson Correlation loss. They also found that using augmented data (translated tweets) generally improved performance, especially for zero-shot languages.

Limitations and Future Work

The authors acknowledge the relatively small size of their dataset, which limited the extent of their experimentation with different model configurations. They also note that the presence of profane words in the original dataset might have biased the model towards those linguistic features, potentially affecting its performance on tweets without such language. Future work could involve exploring more advanced data augmentation techniques, investigating the impact of different model architectures beyond XLM-T, and potentially curating a larger, more balanced dataset for multilingual intimacy analysis.

Summary updated

BibTeX
@inproceedings{rostamkhani2023rozam,
  author = {Mohammadmostafa Rostamkhani and Ghazal Zamaninejad and Sauleh Eetemadi},
  title = {{ROZAM at SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure