I2C-Huelva at SemEval-2023 Task 9: Analysis of Intimacy in Multilingual Tweets Using Resampling Methods and Transformers

Abel Pichardo Estevez, Jacinto Mata Vázquez, Victoria Pachón Álvarez, Nordin El Balima CorderoACL 2023

TL;DR

This paper tackles the challenge of quantifying "intimacy" in multilingual tweets, a nuanced linguistic task. The authors propose a transfer-learning approach using pre-trained transformer models, crucially addressing the dataset's severe class imbalance with novel oversampling and undersampling techniques. Their submission achieved a respectable Pearson's r of 0.497 on the official test set, highlighting the effectiveness of their data balancing strategies.

What Problem Does This Paper Address?

Understanding and quantifying human emotions and social dynamics in text is a cornerstone of natural language processing. This paper focuses on "intimacy," a complex and subjective aspect of human communication, specifically within the context of social media. The SemEval-2023 Task 9 aimed to automatically detect and quantify the level of intimacy expressed in tweets across six languages. This is challenging because intimacy is expressed subtly, often through a combination of word choice, emoticons, and implicit social cues, making it difficult for models to grasp. Furthermore, the provided dataset was heavily imbalanced, with significantly fewer examples of higher intimacy levels, posing a major hurdle for training accurate predictive models.

Methods

The core of the proposed solution lies in a transfer-learning approach, leveraging pre-trained transformer models. Specifically, the authors fine-tuned multilingual models like xlm-roberta-base and bert-base-multilingual-uncased.

To combat the dataset's imbalance, they employed two main strategies:

  • Oversampling: This involved increasing the number of minority class examples. A key technique used was "round-trip translation," where tweets were translated to another language and back to create paraphrased versions with similar intimacy levels but different structures. They also explored oversampling specific intimacy ranges (e.g., tweets with labels > 2.5, > 3, > 4) and a more complex range-based duplication strategy.
  • Undersampling: This aimed to reduce the number of majority class instances. They applied random undersampling (RUS) to the original dataset and also to datasets that had already undergone oversampling.

For the final submission, they focused on the best-performing oversampling technique (O4) and the Twitter/twhin-bert-base model. Additionally, for the test phase, which included new languages (Arabic, Korean, Hindi, Dutch), tweets were translated to English before prediction.

Datasets

The primary dataset used was the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis dataset. This dataset contains 9491 tweets in six languages (Chinese, English, Spanish, Portuguese, French, and Italian), with intimacy levels scored on a scale of 1 to 5. The dataset exhibited a significant imbalance, with fewer examples at higher intimacy levels. The test phase expanded to include Arabic, Korean, Hindi, and Dutch.

Key Insights and Ideas

  • Intimacy is a subtle and complex feature: The paper underscores that quantifying intimacy is not straightforward and requires sophisticated NLP techniques.
  • Data imbalance is a critical challenge: The severe imbalance in the dataset significantly impacts model performance, making data augmentation techniques essential.
  • Oversampling can be more effective than undersampling for this task: The results suggest that carefully crafted oversampling strategies, particularly the range-based duplication (O4), yielded better performance than undersampling alone.
  • Multilingual models are a strong baseline, but monolingual models can offer advantages: While multilingual models provide a good starting point, monolingual models showed superior performance for specific languages like English and Spanish, suggesting a potential avenue for improvement.
  • Translation can bridge language gaps: Translating unseen languages to a model's known language is a viable strategy for extending the model's reach.

Key Results

  • The best oversampling technique (O4) achieved a Pearson's r of 0.79 during the development phase.
  • The best undersampling technique (U2) achieved a Pearson's r of 0.78 during the development phase.
  • The final submission, using the Twitter/twhin-bert-base model and the O4 oversampling technique, achieved an overall Pearson's r of 0.497 on the official test dataset.
  • The model achieved the highest Pearson's r for Spanish (0.673) and the lowest for Hindi (0.206) and Korean (0.253) on the test set.
  • The team was ranked 36th overall in the competition.

Limitations and Future Work

The authors acknowledge that the Pearson's r dropped significantly from the development phase (0.827) to the evaluation phase (0.497). They attribute this difference partly to the incorporation of new languages in the test dataset.

For future work, they propose:

  • Adding more features to the tweets, such as the number and type of emoticons and specific adjectives, which they believe are crucial indicators of intimacy.
  • Developing a more specialized "intimacy language vocabulary" to enhance prediction models.

Summary updated

BibTeX
@inproceedings{estevez2023i,
  author = {Abel Pichardo Estevez and Jacinto Mata Vázquez and Victoria Pachón Álvarez and Nordin El Balima Cordero},
  title = {{I2C-Huelva at SemEval-2023 Task 9: Analysis of Intimacy in Multilingual Tweets Using Resampling Methods and Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure