TL;DR
This paper presents a system for analyzing intimacy in multilingual tweets, leveraging a pre-trained language model and data augmentation via machine translation. The system achieved competitive results in the SemEval-2023 Task 9, ranking 4th for unseen languages and 16th for seen languages, demonstrating the effectiveness of its approach in a challenging cross-lingual setting.
What Problem Does This Paper Address?
Understanding human relationships is a fundamental aspect of social science, and linguistic intimacy plays a crucial role in how we express and perceive these connections. Computational linguistics offers powerful tools to analyze these nuances, but doing so across multiple languages presents significant challenges. Specifically, the expression of intimacy varies greatly across cultures and languages, and the limited availability of labeled data for many languages exacerbates this difficulty. This paper tackles the problem of building a robust multilingual system for intimacy analysis on social media, specifically Twitter, which is rich in diverse social interactions. The goal is to develop a model that can accurately predict intimacy scores across various languages, even those not seen during training.
Methods
The proposed system is built upon a modular architecture consisting of three main components:
- Text Encoder: The system utilizes TwHIN-BERT, a multilingual tweet language model pre-trained on a massive dataset of 7 billion tweets across over 100 languages. This model is chosen for its ability to capture rich semantic information and social engagements within tweets, incorporating a Masked Language Model (MLM) loss objective along with an additional loss term for understanding Twitter's heterogeneous information network.
- Regression Model: A two-layer fully-connected neural network takes the high-dimensional representations from the text encoder and projects them to a single intimacy score. To combat overfitting, especially with limited training data, the model incorporates a Dropout layer and a ReLU activation layer.
- Data Augmentation Module: To address the challenge of low-resource scenarios, the system employs a multilingual data augmentation strategy. This involves using off-the-shelf neural machine translation (NMT) models (based on the Transformer architecture) to translate existing tweets from a source language into multiple target languages. These translated tweets are then added to the training corpus, effectively increasing the amount of available data and exposing the model to more linguistic variations.
- Model Ensemble: To further enhance performance, the system leverages an ensemble of models. For each language, the model with the highest performance score at each epoch is selected. For unseen languages, the checkpoint with the highest average Pearson score across all languages is chosen.
Datasets
- Training Data: The system was trained on 12,000 tweets in six languages: English, Chinese, French, Spanish, Italian, and Portuguese.
- Test Data: The test set comprised 2,000 tweets in four languages not included in the training data: Korean, Dutch, Hindi, and Arabic.
- Parallel Corpus: Neural machine translation models were trained on the OPUS corpus.
Key Insights and Ideas
- Data Augmentation is Crucial for Low-Resource Languages: The paper highlights that translating existing data into new languages significantly boosts performance, particularly for languages that were not part of the original training set. This is a key strategy for overcoming data scarcity in multilingual NLP tasks.
- Cultural Nuances of Intimacy Pose a Challenge: The varying performance across languages, even within the "seen" set, suggests that intimacy is deeply intertwined with cultural context. Differences in how emotions and intimacy are expressed across cultures present inherent difficulties for computational models.
- Subjectivity in Annotation Matters: The authors acknowledge that the data and intimacy labels themselves can be subject to distribution shifts due to the subjective nature of human annotation, which can impact model performance.
- Ensembling Improves Robustness: By selecting the best-performing model for each language and combining them, the system achieves a more robust and generalized performance across the diverse linguistic landscape.
Key Results
- The system achieved a 4th place ranking for languages unseen in the training data and a 16th place ranking for languages seen in the training data in the SemEval-2023 Task 9.
- In-domain performance was strongest in Spanish and weakest in Italian.
- For unseen languages, the model showed good generalization to Dutch, while its performance in Hindi was the weakest, attributed to Hindi being a lower-resource language in the encoder's training corpus and having fewer cultural connections with the training languages.
- Translation data augmentation generally improved performance, especially for Chinese and Italian, though it led to a slight drop in English scores.
Limitations and Future Work
- Computational Constraints: The authors mention that due to computational limitations, they were unable to explore alternative, more powerful multilingual models like mT5 and M2M-100 for data augmentation, which could potentially further improve performance.
- Cultural Context Modeling: While the system shows promise, a deeper integration of cultural context into the model's understanding of intimacy could lead to more nuanced and accurate predictions.
- Exploring Alternative Augmentation Techniques: The paper suggests that exploring different methods for generating silver data, beyond simple translation, could be a fruitful direction for future research.
Summary updated