TL;DR
This paper tackles the challenge of predicting textual intimacy across 10 languages by addressing severe class imbalance in the dataset. They propose a novel approach combining machine translation for data augmentation and a "focal MSE loss" function to better capture subtle nuances in intimacy. Their method achieves strong performance, demonstrating the effectiveness of these strategies for multilingual intimacy analysis.
What Problem Does This Paper Address?
Understanding the "intimacy" of communication – how close people feel based on their language – is a fascinating area with applications in social psychology and understanding human interaction. While previous work has explored this, it often relied on specific question formats. SemEval-2023 Task 9 aims to broaden this by predicting intimacy scores for tweets across 10 languages. The core challenge here is twofold: first, quantifying intimacy computationally, and second, doing so effectively across diverse languages. A significant hurdle identified by the authors is the severe class imbalance in the provided dataset, where certain intimacy levels are vastly underrepresented. This imbalance can hinder model training and lead to biased predictions, making it difficult for models to learn from the less frequent, potentially more nuanced, intimacy levels.
Methods
The authors leverage a pre-trained multilingual language model, XLM-RoBERTa (XLM-R), as their base for this regression task. To combat the class imbalance, they employ two key strategies:
- Data Augmentation via Machine Translation: They use Google Translate to translate tweets from underrepresented classes into other languages. This effectively "upsamples" the minority classes, creating a more balanced dataset. They also translate some existing data into the four new languages present in the test set to better simulate its distribution.
- Focal MSE Loss: Standard Mean Squared Error (MSE) loss can be dominated by "easy" samples (those with clear intimacy scores). To give more weight to "hard" samples (those with less clear or extreme intimacy scores), they introduce a "focal MSE loss." This loss function incorporates a modulating factor that amplifies the loss for samples where the model's prediction is far from the true label, effectively forcing the model to pay more attention to these challenging cases.
Datasets
The primary dataset used is the official SemEval-2023 Task 9 dataset, which contains tweets in 10 languages (English, Spanish, Italian, Portuguese, French, Chinese, Hindi, Arabic, Dutch, and Korean). The dataset is annotated with intimacy scores ranging from 1 to 5, where higher scores indicate greater intimacy. The authors augment this dataset using machine translation, resulting in a significantly larger training set (22,279 items).
Key Insights and Ideas
- Class Imbalance is a Major Hurdle: The paper strongly emphasizes that class imbalance is not just a minor inconvenience but a significant obstacle that can severely impact model performance in this task.
- Machine Translation as a Powerful Augmentation Tool: Using machine translation to generate synthetic data proved effective in balancing the dataset and improving model generalization, especially for underrepresented classes.
- Focal Loss for Regression is Promising: Adapting the focal loss concept, originally for classification, to a regression task (focal MSE loss) effectively highlights hard-to-predict samples, leading to better performance. The intuition is that samples with large prediction errors (discrepancies) are more informative for learning.
- Multilingual Models are Key: XLM-R's ability to handle multiple languages out-of-the-box is crucial for tackling this multilingual task.
Key Results
The proposed model, combining XLM-R with data augmentation and focal MSE loss, achieved strong results:
- On the raw dataset, their focal MSE loss improved Pearson's CC from 0.576 to 0.675 compared to standard MSE loss (without cross-validation).
- On the augmented dataset, their model achieved a Pearson's CC of 0.885 (on English & Portuguese) and 0.858 (on Chinese & Arabic) without cross-validation, and even higher scores with 5-fold and 10-fold cross-validation (e.g., 0.861 with 10-fold).
- On the official test set, they achieved an average score of 0.65 on the 6 languages present in the training set (ranking 31st) and 0.38 on the 4 new languages (ranking 27th).
Limitations and Future Work
The authors acknowledge that overfitting on the augmented dataset still exists. They suggest that future work could explore more advanced techniques to mitigate this limitation. Additionally, while their focal MSE loss shows promise, further research into its optimal application and hyperparameter tuning could be beneficial. The varying performance across language pairs also suggests that language-specific nuances might require further investigation.
Summary updated