TL;DR
This paper tackles the challenge of automatically detecting "intimacy" in multilingual tweets, a complex task due to language diversity and the subtle nature of intimacy. The authors propose a system that leverages a multilingual pre-trained model (XLM-T) and enhances it with adversarial training, data augmentation, and a novel ordinal regression loss function, achieving competitive results in a SemEval competition.
What Problem Does This Paper Address?
Understanding and quantifying the level of intimacy in human communication is crucial for various applications, from social science research to building more empathetic AI systems. This paper focuses on the specific challenge of mining intimacy in tweets, which are notoriously short, informal, and often context-dependent. The SemEval-2023 Task 9 introduces a further layer of complexity by requiring models to handle multilingual data and perform zero-shot cross-lingual learning. This means the model must predict intimacy in languages it hasn't explicitly been trained on, a significant hurdle given the diverse ways intimacy is expressed across cultures and languages. The task also involves a continuum scale for intimacy (1-5), requiring models to capture nuanced differences rather than just binary classifications.
Methods
The core of the proposed system is the XLM-T model, a powerful pre-trained language model designed for multilingual text. To adapt XLM-T for intimacy analysis, the authors employ several key techniques:
- Adversarial Training (Fast Gradient Method - FGM): This technique introduces small, carefully crafted perturbations to the input embeddings. The goal is to make the model more robust to minor variations in text, thereby improving its generalization capabilities and preventing overfitting.
- Data Augmentation: To address the scarcity of data and the zero-shot cross-lingual challenge, the authors implement two levels of data augmentation:
- Word-level Substitution: Randomly masking and substituting word pieces using a masked language model (BERT) to introduce noise and improve robustness.
- Sentence-level Translation: Translating tweets into both seen and unseen languages using Azure Translator API. This enriches the training data with diverse linguistic expressions of intimacy.
- Ordinal Regression Loss: Instead of a standard regression loss (like MSE), the authors adopt an ordinal regression loss. This is inspired by the fact that the intimacy labels (1-5) have an inherent order. This loss function penalizes predictions that are further away from the true label more heavily, effectively learning the ordered relationship between intimacy levels. They define boundary values to model the probability of each label.
- Multilingual Cross-Validation Training Strategy: To better handle the multilingual nature of the task and improve zero-shot performance, they employ a specialized training strategy. They randomly split 10% of the training data as a validation set and rotate which of the six seen languages is designated as the "unseen" language for validation every 13 epochs. This ensures the model is consistently tested on its ability to generalize to new languages.
Datasets
The primary dataset used is the MINT dataset introduced by Pei et al. (2023) for SemEval-2023 Task 9. This dataset contains 13,372 tweets across 10 languages, with intimacy labels on a 1-5 scale. The training set includes six languages (English, Spanish, Italian, Portuguese, French, and Chinese), while the test set comprises ten languages, including four unseen languages (Hindi, Arabic, Dutch, and Korean).
Key Insights and Ideas
- Ordinal Regression is Key for Ordered Labels: The paper strongly suggests that treating intimacy labels as ordered categories (ordinal regression) is more effective than treating them as continuous values or independent classes. This aligns with the inherent nature of the 1-5 scale.
- Data Augmentation is a Double-Edged Sword for Zero-Shot: While translation augmentation and word-level noise on seen languages improve performance, adding noise to unseen language data can actually hurt performance. This highlights the risk of introducing distribution errors when augmenting data for languages the model has little exposure to.
- Multilingual Pre-training is Powerful, but Needs Careful Fine-tuning: XLM-T provides a strong foundation, but its effectiveness is significantly boosted by tailored optimization techniques like adversarial training and the specific ordinal regression loss.
- Robustness through Adversarial Training: Adversarial training demonstrably improves the model's ability to handle slight variations in input, leading to better overall performance.
Key Results
The proposed model achieved a Pearson correlation score of 0.7029 on the seen language section and 0.4359 on the unseen language section. This placed them 14th out of 54 participating teams overall and 10th on predicting languages not in the training data. The paper also shows that the ordinal regression loss significantly outperforms MSE loss, and adversarial training provides a noticeable improvement.
Limitations and Future Work
The authors acknowledge that for low-resource languages, the performance of multilingual models can hit a bottleneck. They suggest that future work should focus on exploring the relationships between different linguistic features and building stronger multilingual multitasking systems for intimacy analysis. The paper also notes that while data augmentation is beneficial, care must be taken when applying it to unseen languages to avoid introducing detrimental distribution errors.
Summary updated