Lazybob at SemEval-2023 Task 9: Quantifying Intimacy of Multilingual Tweets with Multi-Task Learning

Mengfei Yuan, Cheng ChenACL 2023

TL;DR

This paper presents a winning approach for quantifying intimacy in multilingual tweets, achieving first place in SemEval-2023 Task 9. They leveraged a novel multi-task learning framework, domain-specific pre-training on Twitter data, and advanced fine-tuning techniques to achieve state-of-the-art performance across 10 languages, including zero-shot settings.

What Problem Does This Paper Address?

Understanding intimacy in human communication is crucial for various applications, from social sciences to human-computer interaction. However, quantifying intimacy, especially in the context of short, informal, and multilingual social media text like tweets, presents a significant challenge. Existing methods often struggle with the nuances of language, cultural differences, and the limited availability of annotated data, particularly for low-resource languages. This paper tackles the problem of accurately predicting the level of intimacy in tweets across ten different languages, aiming to build a robust and generalizable model.

Methods

The authors employed a multi-task learning (MTL) framework to simultaneously train their model on data from all available languages. This approach helps the model learn shared representations and improve generalization, especially for languages with less data. Key techniques include:

  • Domain-Specific Pre-training: They further pre-trained language models (specifically TwHin-BERT) on a massive corpus of 10 million crawled Twitter, book, movie, and Reddit data. This "domain pre-training" helps the models better understand the specific linguistic characteristics of social media text.
  • Multi-Task Learning for Regression and Classification: To address the imbalanced label distribution and improve prediction accuracy, they converted the intimacy scores into pseudo-labels for a classification task. The model was then trained jointly on both the regression (predicting a continuous intimacy score) and classification (predicting intimacy categories) tasks.
  • Advanced Fine-tuning Strategies:
    • Weighted Random Sampler: Used to give more importance to certain samples during training, potentially mitigating the impact of noisy data.
  • Attention Mechanism: Applied to the text embedding output to allow the model to focus on the most relevant parts of the input for predicting intimacy.
  • Multi-sample Dropout: A regularization technique to improve model generalization, particularly for classification tasks within the MTL setup.
  • Adversarial Weight Perturbation (AWP): A contrastive learning technique to enhance the robustness of learned representations.
  • Group-Layer-wise Learning Rate Decay (GLLRD): A method to adjust learning rates across different layers of the network, helping to prevent overfitting and improve training efficiency.
  • Data Augmentation for Zero-Shot Languages: For languages without training data (zero-shot languages), they used bidirectional translation to augment the English training data, aiming to improve performance on these underrepresented languages.
  • Ensemble Method: The final submission combined results from 25 different Multi-learning Task models trained using TwHin-BERT.

Datasets

  • Official Training Data: Provided by SemEval-2023 Task 9, containing tweets in 10 languages. Six languages had training data (English, Spanish, Portuguese, Italian, French, Chinese), and four were zero-shot (Korean, Arabic, Hindi, Dutch).
  • External Data for Pre-training: 10 million crawled data from Twitter, books, movies, and Reddit. This was used to further pre-train the TwHin-BERT model.

Key Insights and Ideas

  • Multi-task learning is crucial for imbalanced regression tasks: By combining regression with a classification objective derived from the labels, the model achieved a more accurate prediction distribution, closer to the annotated data.
  • Domain-specific pre-training significantly boosts performance: Pre-training on a large corpus of Twitter data (like with TwHin-BERT) is more effective than using general-purpose language models for this task.
  • Data augmentation via translation helps zero-shot performance: Translating training data into English improved the model's ability to predict intimacy in languages it hadn't seen during training.
  • Fine-tuning techniques matter: Advanced strategies like AWP and GLLRD contribute to model robustness and better generalization.

Key Results

The authors' submission achieved the first place overall score in the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis, with a Pearson R of 0.6160 on the official test set, outperforming 45 other teams. Their TwHin-BERT model, trained using multi-task learning, provided the best performance across most languages. Data augmentation and multi-task learning were particularly effective for improving prediction accuracy, especially for zero-shot languages.

Limitations and Future Work

  • Limited Sample Size for External Data: Due to time and cost constraints, they only sampled 10 million out of 80.5 million available external data for pre-training.
  • Translation Noise: While translation augmentation helped zero-shot performance, it might have introduced noise, and the results were not as good as using the multi-language strategy directly.
  • Future Work: The authors suggest exploring feature engineering, tuning more deep learning tricks, and considering pre-training models with more relevant content to further improve robustness and prediction accuracy for specific conversation types.

Summary updated

BibTeX
@inproceedings{yuan2023lazybob,
  author = {Mengfei Yuan and Cheng Chen},
  title = {{Lazybob at SemEval-2023 Task 9: Quantifying Intimacy of Multilingual Tweets with Multi-Task Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure