YNU-HPCC at SemEval-2023 Task 9: Pretrained Language Model for Multilingual Tweet Intimacy Analysis

Qisheng Cai, Jin Wang, Xuejie ZhangACL 2023

TL;DR

This paper introduces a novel approach for analyzing intimacy in multilingual tweets by fine-tuning the XLM-T language model with a bidirectional GRU (BiGRU) layer enhanced by attention and memory mechanisms. The proposed XLMT-GRU model significantly improves performance on both seen and unseen languages, demonstrating the effectiveness of combining pre-trained multilingual models with recurrent neural networks for nuanced social media analysis.

What Problem Does This Paper Address?

Analyzing the intimacy of social media posts is crucial for understanding social dynamics and norms. However, existing methods often struggle with the complexities of multilingual text, especially code-mixed tweets that blend multiple languages and incorporate informal elements like slang, emojis, and usernames. These unconventional texts pose a significant challenge for standard NLP models, leading to information loss or misinterpretation. The SemEval-2023 Task 9 (MTIA) specifically aims to address this by developing systems that can quantitatively analyze tweet intimacy across six languages, with a particular focus on zero-shot learning for unseen languages.

Methods

The authors propose a model called XLMT-GRU that builds upon the XLM-T pre-trained language model. XLM-T is chosen because it's specifically trained on Twitter data, making it more adept at handling informal language and symbolic representations compared to general-purpose multilingual models.

The core innovation lies in how they process the output of XLM-T:

  • XLM-T Layer: This layer takes the preprocessed tweets and generates sentence embeddings.
  • Bidirectional GRU (BiGRU): Two layers of BiGRU are stacked on top of XLM-T. BiGRUs are effective at capturing long-term dependencies in sequential data by processing information from both past and future time steps. This is crucial for understanding the context and nuances of a tweet.
  • Attention and Memory Mechanisms: While not explicitly detailed as separate modules in the architecture diagram, the authors state their approach "combines attention and memory mechanisms," inspired by human cognition. This likely refers to how the GRU layers, particularly with their gating mechanisms, implicitly handle attention to relevant parts of the sequence and maintain memory of past states.
  • Linear Layer: A final linear layer maps the BiGRU output to a single scalar value representing the intimacy score (1-5).
  • Loss Function: The Mean Squared Error (MSE) is used to train the model, minimizing the difference between predicted and actual intimacy scores.

For preprocessing, they focus on removing usernames (tokens starting with "@") to reduce bias and improve fairness, while carefully handling the "@" character itself.

Datasets

The primary dataset used is the MINT dataset, provided by Pei et al. (2022). This dataset contains 13,384 tweets in ten languages. For training, the authors use the first six languages (English, Spanish, Portuguese, Italian, French, Chinese), which are considered "seen" languages. The remaining four languages (Hindi, Korean, Dutch, Arabic) are used for "unseen" or zero-shot learning evaluation. The dataset is split into training and development sets with an 8:2 ratio.

Key Insights and Ideas

  • XLM-T's Advantage: Pre-training on Twitter data (like XLM-T) is significantly more beneficial for analyzing tweet intimacy than general multilingual models (like XLM-R), especially for handling informal language and emojis.
  • GRU for Context: Stacking bidirectional GRU layers on top of powerful pre-trained embeddings effectively captures sequential dependencies and context, leading to improved intimacy prediction.
  • Multilingual Nuances: The performance on unseen languages highlights linguistic similarities. Dutch and Portuguese, being Romance languages, show strong transfer learning. Arabic also performs relatively well despite script differences, suggesting shared semantic representations. Hindi and Korean, with their distinct grammatical structures, present greater challenges for zero-shot learning.
  • Attention/Memory is Key: The integration of attention and memory mechanisms (implicitly within the GRU architecture) is highlighted as a critical factor for boosting performance, especially on smaller datasets where GRUs can converge faster and outperform simpler linear models.

Key Results

  • Overall Performance: The XLMT-GRU model achieved 13th place overall in the SemEval-2023 Task 9 leaderboard.
  • Seen Languages: The model showed significant improvements in prediction accuracy for English and Spanish. It generally outperformed the baseline XLM-T model, achieving a Pearson's r score of 0.7109 on the test set.
  • Unseen Languages: The model demonstrated strong performance on Dutch and Arabic, ranking fourth in these categories. While performance on Hindi and Korean regressed compared to the baseline, the overall zero-shot learning capability was enhanced.
  • Comparison: XLMT-GRU consistently outperformed its counterparts (XLM-R and XLMR-GRU) across most languages, underscoring the benefits of XLM-T and the GRU integration.

Limitations and Future Work

  • Hindi Performance: The paper notes a regression in performance for Hindi on unseen languages. The authors attribute this to significant differences in grammatical rules and writing systems between Hindi and the seen languages.
  • Chinese Performance: Performance on Chinese in seen languages was somewhat weakened, possibly due to the relatively small size of Chinese text in the Twitter dataset compared to English.
  • Future Work: The authors plan to explore integrating other models, such as TreeLSTM, to further complement the current model's shortcomings, particularly for languages like Hindi. They also aim to investigate how to better integrate attention and memory mechanisms to boost performance on small datasets.

Summary updated

BibTeX
@inproceedings{cai2023ynu,
  author = {Qisheng Cai and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2023 Task 9: Pretrained Language Model for Multilingual Tweet Intimacy Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure