TL;DR
This paper tackles the challenge of understanding "intimacy" in multilingual tweets, a nuanced aspect of human communication. They achieve strong results by combining domain-adapted pre-trained language models with a novel translation-based data augmentation strategy, significantly improving performance on unseen languages.
What Problem Does This Paper Address?
Understanding the subtle nuances of human communication, like "intimacy," in text is a complex task for AI. This becomes even more challenging when dealing with multilingual data, as different languages and cultures express intimacy in diverse ways. The SemEval-2023 Task 9 specifically aimed to quantify intimacy in tweets across 10 languages, ranging from 1 (low intimacy) to 5 (high intimacy). This task is crucial for applications like social network analysis, understanding information spread, and analyzing online community dynamics. The core problem is developing a robust system that can accurately assess intimacy across a diverse set of languages, especially those not well-represented in the training data.
Methods
The authors propose a pipeline that leverages several key components:
- Domain-Adapted Pre-trained Models: They experimented with several multilingual transformer models, finding that TwHIN-BERT (trained on a massive Twitter corpus) and XLM-T performed best. For their final system, they used TwHIN-BERT.
- Data Cleaning: Standard tweet preprocessing steps were applied, including replacing URLs with "http", usernames with "@user", and removing extra punctuation and spaces.
- Emoji Representation: Emojis are treated as important signals of intimacy. They employ a two-phase approach:
- Extracting emojis and using
emoji2vecto get fixed-length vector representations, which are then mean-pooled and concatenated with sentence embeddings. - Converting emojis to their textual descriptions using the
emojiPython package and incorporating these into the text. - Translation-Based Data Augmentation: This is a core innovation. To improve performance on unseen languages (Hindi, Korean, Dutch, Arabic), they translate tweets from seen languages (English, Spanish, Portuguese, Italian, French) into these unseen languages. This helps the model learn cross-lingual transfer. They also experimented with translating unseen languages to English for inference, but this proved less effective.
Datasets
- Training Data: The Multilingual Intimacy Analysis (MINT) dataset, containing 9491 tweets in 6 languages (English, Spanish, Portuguese, Italian, French, Chinese).
- Test Data: The MINT dataset's test split, with 13697 tweets across 10 languages, including the 6 seen languages and 4 unseen languages (Hindi, Korean, Dutch, Arabic).
Key Insights and Ideas
- Translation is Key for Unseen Languages: Augmenting the training data with translations into unseen languages significantly boosts performance, demonstrating the power of this technique for cross-lingual transfer.
- Emojis Carry Significant Intimacy Signals: Incorporating emoji representations (both vector embeddings and textual descriptions) is crucial for accurate intimacy prediction. Simply removing them leads to a noticeable drop in performance.
- Domain-Specific Models Matter: Pre-training language models on large, domain-specific corpora like Twitter (e.g., TwHIN-BERT) yields better results than general-purpose multilingual models for this task.
- Data Cleaning Offers Minor Gains: While important, aggressive data cleaning had a less pronounced impact compared to translation augmentation and emoji handling.
Key Results
The authors' submitted system achieved:
- Overall Leaderboard: 10th place with a Pearson's r score of 0.592.
- Unseen Languages Leaderboard: 3rd place with a Pearson's r score of 0.485.
Ablation studies revealed that:
- Removing data cleaning resulted in a minor performance drop.
- Not using emoji representations led to a significant drop in scores for seen languages and a minor drop overall.
- Translation-based augmentation was critical, with approaches that excluded it showing substantial performance degradation, especially for unseen languages.
Limitations and Future Work
- Translation Errors: The authors note that translating unseen languages to English for inference resulted in a greater performance drop, possibly due to propagation of translation errors or differences in style and distribution between original and translated tweets.
- Hyperparameter Sensitivity: They mention that transformer-based models are prone to variance, and fine-tuning hyperparameters can lead to significant gains.
- Future Work: While not explicitly detailed, the success of domain-adapted models and translation augmentation suggests further exploration in these areas, potentially with more advanced translation techniques or by investigating other domain-specific features. The paper also hints at the potential for simple deep learning models to achieve competitive results when paired with strong domain-adapted pre-trained transformers.
Summary updated