SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis

Jiaxin Pei, Vítor Silva, Maarten W. Bos, Yozen Liu, Leonardo Neves, David Jurgens, Francesco Barbieri

Introduction

Intimacy is a fundamental social dimension of human interaction, yet it remains significantly under-researched in the field of Natural Language Processing (NLP). While existing research extensively covers sentiment analysis and offensiveness, the subtle nuances of how people share personal feelings or "self-disclose" in daily life are often overlooked. This paper introduces MINT, the first large-scale multilingual dataset designed to quantify intimacy in social media text, covering 13,372 tweets across 10 languages.

The challenge of quantifying intimacy is twofold. First, it is highly subjective; what one individual considers an intimate disclosure, another might find mundane. Second, there is significant cross-cultural variance, as intimacy is expressed and perceived differently across various languages and cultures. To address these challenges, the authors organized SemEval-2023 Task 9, a global competition where 45 teams competed to predict intimacy levels across diverse linguistic contexts.

The MINT Dataset and Methodology

The researchers developed a rigorous pipeline to create the MINT (Multilingual INTimacy) dataset. To ensure the dataset wasn't skewed toward "not intimate" content, they employed a "bucket sampling" technique, over-sampling tweets likely to contain higher levels of intimacy based on a preliminary model.

Each of the 13,372 tweets was rated by seven native speakers on a 5-point Likert scale (1: Not intimate at all, 5: Very intimate). Strict quality controls, including demographic balancing and attention checks, were implemented to ensure reliable human-aggregated scores.

Distribution of intimacy scores across the 10 languages in the MINT dataset, showing a general skew toward lower intimacy levels with variation across cultures.

The dataset includes training data for English, Spanish, French, Portuguese, Italian, and Chinese. Crucially, it also includes "zero-shot" test languages—Dutch, Korean, Hindi, and Arabic—to evaluate how well models generalize to languages they were not specifically trained on for this task.

Key Results and Performance

The competition saw 45 teams from over 20 countries. The top-performing team, LAZYBOB, achieved an overall Pearson’s $r$ (correlation) of 0.616. A key finding was that domain-specific pre-training is vital; the XLM-T model (pre-trained on Twitter data) consistently outperformed the standard XLM-RoBERTa, suggesting that understanding the "dialect" of social media (emojis, hashtags, informal grammar) is essential for social tasks.

However, the results also highlighted a significant performance gap between languages. While models performed exceptionally well on Spanish ($r \approx 0.78$) and English, they struggled with zero-shot predictions for culturally and linguistically distinct languages like Hindi ($r \approx 0.28$) and Korean ($r \approx 0.41$).

Model performance (Pearson’s r) across different languages, highlighting the significant drop in zero-shot performance for Hindi and Korean compared to Western European languages.

Insights and Future Directions

The performance disparity suggests that current multilingual models still struggle with "cross-lingual transfer" for subtle social nuances. The "Chinese Outlier" effect also showed that performance on Chinese tweets had lower correlation with Western European languages, indicating distinct cultural expressions of intimacy.

While most teams relied on traditional fine-tuning for regression, the authors suggest that future work should explore more sophisticated prompting strategies. Furthermore, because intimacy is inherently subjective, the authors plan to release individual labels and demographic data. This will allow researchers to move beyond mean scores and model how different groups—based on gender or age—perceive intimacy differently, ultimately leading to more socially aware dialogue systems and content moderation tools.

Summary updated

Abstract

Intimacy is an important social aspect of language. Computational modeling of intimacy in language could help many downstream applications like dialogue systems and offensiveness detection. Despite its importance, resources and approaches on modeling textual intimacy remain rare. To address this gap, we introduce MINT, a new Multilingual intimacy analysis dataset covering 13,372 tweets in 10 languages including English, French, Spanish, Italian, Portuguese, Korean, Dutch, Chinese, Hindi, and Arabic along with SemEval 2023 Task 9: Multilingual Tweet Intimacy Analysis. Our task attracted 45 participants from around the world. While the participants are able to achieve overall good performance on languages in the training set, zero-shot prediction of intimacy in unseen languages remains challenging. Here we provide an overview of the task, summaries of the common approaches, and potential future directions on modeling intimacy across languages. All the relevant resources are available at https: //sites.google.com/umich.edu/ semeval-2023-tweet-intimacy.
BibTeX
@inproceedings{pei2023semeval,
  author = {Jiaxin Pei and Vítor Silva and Maarten W. Bos and Yozen Liu and Leonardo Neves and David Jurgens and Francesco Barbieri},
  title = {{SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis}},
  booktitle = {SemEval @ ACL},
  year = {2023},
  doi = {10.18653/v1/2023.semeval-1.309},
}
Figure