DUTH at SemEval-2023 Task 9: An Ensemble Approach for Twitter Intimacy Analysis

Giorgos Arampatzis, Vasileios Perifanis, Symeon Symeonidis, Avi ArampatzisACL 2023

TL;DR

This paper tackles the challenge of automatically analyzing "intimacy" in multilingual tweets, a nuanced aspect of human communication. The DUTH team developed an ensemble model that leverages pre-trained transformer embeddings (like XLM-T) and simpler classifiers (MLP, CatBoost) to achieve competitive results without the need for computationally expensive fine-tuning of large transformer models.

What Problem Does This Paper Address?

Understanding the emotional closeness or "intimacy" between individuals is crucial for analyzing user behavior, sentiment, and preferences, with applications in social media, healthcare, and marketing. However, analyzing intimacy in natural language, especially across multiple languages, presents a significant challenge due to its subjective and context-dependent nature. While resources for sentiment analysis are abundant, datasets and computational tools for intimacy analysis are scarce. This paper addresses this gap by participating in SemEval-2023 Task 9, which aims to develop computational models for multilingual tweet intimacy analysis using a newly released dataset.

Methods

The DUTH team's approach is built around two core ideas: leveraging powerful pre-trained transformer embeddings and using an ensemble of simpler, efficient models.

  1. Text-to-Feature Representation: Instead of fine-tuning large transformer models (which is time-consuming), they explored various methods to convert tweets into numerical features. These included traditional methods like CountVectorizer and TF-IDF, and more advanced ones like Doc2Vec, XLM-R, and XLM-T. XLM-T, a transformer specifically designed for multilingual tweets, proved to be the most effective for generating rich text embeddings.
  2. Machine Learning Models: They experimented with several regression models, including Linear Regression, k-Nearest Neighbors, Random Forest, XGBoost, CatBoost, and Multilayer Perceptron (MLP). They found that MLP and CatBoost consistently performed well.
  3. Ensemble Approach: To further boost performance, they created an ensemble of models. Their final system combined three MLPs with different architectures and a CatBoost model. A weighted voting mechanism, determined through a grid search, was used to combine the predictions of these individual models.
  4. Pre-processing: The paper also investigated the impact of various pre-processing techniques (e.g., lowercasing, removing HTML tags, converting numbers to words). Their findings indicated that these techniques had a minimal impact on the overall regression accuracy for this specific task.

Datasets

The primary dataset used is the multilingual dataset released for SemEval-2023 Task 9. This dataset consists of tweets in ten languages: English, Spanish, French, Portuguese, Italian, Chinese, Hindi, Korean, Dutch, and Arabic. The training set contained 9,491 annotated tweets, and the test set had 13,697 tweets. The training set specifically included six languages: English, Spanish, Portuguese, Italian, French, and Chinese.

Key Insights and Ideas

  • Transformer Embeddings Shine: Pre-trained transformer embeddings, particularly XLM-T, provide a significant advantage over traditional text representation methods (CountVectorizer, TF-IDF, Doc2Vec) for multilingual intimacy analysis.
  • Efficiency Over Fine-Tuning: Fine-tuning large transformer models is not always necessary. Using their embeddings with simpler models like MLPs and CatBoost can achieve comparable or even better results with significantly less computational cost and training time.
  • Pre-processing is Less Critical Than Expected: For this specific task and with advanced embeddings, extensive pre-processing steps had a surprisingly small impact on the final performance. This suggests that the power of the embeddings themselves carries most of the information.
  • Ensembling for Robustness: Combining multiple well-performing models (MLPs and CatBoost) through weighted voting further improved the accuracy and robustness of the final prediction.
  • Imbalanced Data is a Challenge: The dataset exhibits a skewed distribution of intimacy levels, with most tweets being less intimate. While the LogCosh loss function was used to mitigate this, it's acknowledged as a limitation that could compromise results.

Key Results

  • The DUTH team achieved a rank of 7 in the overall SemEval-2023 Task 9 competition.
  • They ranked 25th on the languages seen during training and 2nd on cross-language transfer learning.
  • XLM-T and XLM-R embeddings, when used with MLP and CatBoost, showed a remarkable improvement in Pearson's r correlation (up to ~76.5% improvement compared to TF-IDF).
  • The final ensemble model, using a weighted combination of three MLPs and CatBoost with XLM-T embeddings, achieved the best performance.
  • MLP and CatBoost models consistently outperformed other machine learning models across various vectorizers.

Limitations and Future Work

  • Imbalanced Regression Task: The paper acknowledges that the imbalanced nature of the intimacy dataset could compromise the quality of the results. Future work should focus on addressing this imbalance more effectively to achieve higher-quality outcomes.
  • Cross-Language Performance Variation: While the model performed well overall, accuracy scores varied significantly across languages, with Chinese and Hindi showing particularly challenging results. Further investigation into language-specific nuances and more robust cross-lingual strategies could be beneficial.
  • Computational Cost of Transformers: Although the paper advocates for using embeddings without fine-tuning, the initial generation of these embeddings from large transformer models still requires significant computational resources.

Summary updated

BibTeX
@inproceedings{arampatzis2023duth,
  author = {Giorgos Arampatzis and Vasileios Perifanis and Symeon Symeonidis and Avi Arampatzis},
  title = {{DUTH at SemEval-2023 Task 9: An Ensemble Approach for Twitter Intimacy Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure